<!DOCTYPE html>
<html lang="en">
<!-- Produced from a LaTeX source file.  Note that the production is done -->
<!-- by a very rough-and-ready (and buggy) script, so the HTML and other  -->
<!-- code is quite ugly!  Later versions should be better.                -->
<head>
    <meta charset="utf-8">
    <meta name="citation_title" content="Neural Networks and Deep Learning">
    <meta name="citation_author" content="Nielsen, Michael A.">
    <meta name="citation_publication_date" content="2015">
    <meta name="citation_fulltext_html_url" content="http://neuralnetworksanddeeplearning.com">
    <meta name="citation_publisher" content="Determination Press">
    <meta name="citation_fulltext_world_readable" content="">
    <link rel="icon" href="http://neuralnetworksanddeeplearning.com/nnadl_favicon.ICO" />
    <title>Neural networks and deep learning</title>
    <script src="assets/jquery.min.js"></script>
    <script type="text/x-mathjax-config">
      MathJax.Hub.Config({
        tex2jax: {inlineMath: [['$','$']]},
        "HTML-CSS": 
          {scale: 92},
        TeX: { equationNumbers: { autoNumber: "AMS" }}});
    </script>
    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>


    <link href="assets/style.css" rel="stylesheet">
    <link href="assets/pygments.css" rel="stylesheet">
    <link rel="stylesheet" href="https://code.jquery.com/ui/1.11.2/themes/smoothness/jquery-ui.css">

<style>
/* Adapted from */
/* https://groups.google.com/d/msg/mathjax-users/jqQxrmeG48o/oAaivLgLN90J, */
/* by David Cervone */

@font-face {
    font-family: 'MJX_Math';
    src: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); /* IE9 Compat Modes */
    src: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot?iefix') format('eot'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff')  format('woff'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf')  format('opentype'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/svg/MathJax_Math-Italic.svg#MathJax_Math-Italic') format('svg');
}

@font-face {
    font-family: 'MJX_Main';
    src: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); /* IE9 Compat Modes */
    src: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot?iefix') format('eot'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff')  format('woff'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf')  format('opentype'),
    url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/fonts/HTML-CSS/TeX/svg/MathJax_Main-Regular.svg#MathJax_Main-Regular') format('svg');
}
</style>

  </head>
  <body><div class="header"><h1 class="chapter_number">
  <a href="chap6.html">CHAPTER 6</a></h1>
  <h1 class="chapter_title"><a href="chap6.html">Deep learning</a></h1></div><div class="section"><div id="toc"> 
<p class="toc_title"><a href="index.html">Neural Networks and Deep Learning</a></p><p class="toc_not_mainchapter"><a href="about.html">What this book is about</a></p><p class="toc_not_mainchapter"><a href="exercises_and_problems.html">On the exercises and problems</a></p><p class='toc_mainchapter'><a id="toc_using_neural_nets_to_recognize_handwritten_digits_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_using_neural_nets_to_recognize_handwritten_digits" src="images/arrow.png" width="15px"></a><a href="chap1.html">Using neural nets to recognize handwritten digits</a><div id="toc_using_neural_nets_to_recognize_handwritten_digits" style="display: none;"><p class="toc_section"><ul><a href="chap1.html#perceptrons"><li>Perceptrons</li></a><a href="chap1.html#sigmoid_neurons"><li>Sigmoid neurons</li></a><a href="chap1.html#the_architecture_of_neural_networks"><li>The architecture of neural networks</li></a><a href="chap1.html#a_simple_network_to_classify_handwritten_digits"><li>A simple network to classify handwritten digits</li></a><a href="chap1.html#learning_with_gradient_descent"><li>Learning with gradient descent</li></a><a href="chap1.html#implementing_our_network_to_classify_digits"><li>Implementing our network to classify digits</li></a><a href="chap1.html#toward_deep_learning"><li>Toward deep learning</li></a></ul></p></div>
<script>
$('#toc_using_neural_nets_to_recognize_handwritten_digits_reveal').click(function() { 
   var src = $('#toc_img_using_neural_nets_to_recognize_handwritten_digits').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_using_neural_nets_to_recognize_handwritten_digits").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_using_neural_nets_to_recognize_handwritten_digits").attr('src', 'images/arrow.png');
   };
   $('#toc_using_neural_nets_to_recognize_handwritten_digits').toggle('fast', function() {});  
});</script><p class='toc_mainchapter'><a id="toc_how_the_backpropagation_algorithm_works_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_how_the_backpropagation_algorithm_works" src="images/arrow.png" width="15px"></a><a href="chap2.html">How the backpropagation algorithm works</a><div id="toc_how_the_backpropagation_algorithm_works" style="display: none;"><p class="toc_section"><ul><a href="chap2.html#warm_up_a_fast_matrix-based_approach_to_computing_the_output_from_a_neural_network"><li>Warm up: a fast matrix-based approach to computing the output  from a neural network</li></a><a href="chap2.html#the_two_assumptions_we_need_about_the_cost_function"><li>The two assumptions we need about the cost function</li></a><a href="chap2.html#the_hadamard_product_$s_\odot_t$"><li>The Hadamard product, $s \odot t$</li></a><a href="chap2.html#the_four_fundamental_equations_behind_backpropagation"><li>The four fundamental equations behind backpropagation</li></a><a href="chap2.html#proof_of_the_four_fundamental_equations_(optional)"><li>Proof of the four fundamental equations (optional)</li></a><a href="chap2.html#the_backpropagation_algorithm"><li>The backpropagation algorithm</li></a><a href="chap2.html#the_code_for_backpropagation"><li>The code for backpropagation</li></a><a href="chap2.html#in_what_sense_is_backpropagation_a_fast_algorithm"><li>In what sense is backpropagation a fast algorithm?</li></a><a href="chap2.html#backpropagation_the_big_picture"><li>Backpropagation: the big picture</li></a></ul></p></div>
<script>
$('#toc_how_the_backpropagation_algorithm_works_reveal').click(function() { 
   var src = $('#toc_img_how_the_backpropagation_algorithm_works').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_how_the_backpropagation_algorithm_works").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_how_the_backpropagation_algorithm_works").attr('src', 'images/arrow.png');
   };
   $('#toc_how_the_backpropagation_algorithm_works').toggle('fast', function() {});  
});</script><p class='toc_mainchapter'><a id="toc_improving_the_way_neural_networks_learn_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_improving_the_way_neural_networks_learn" src="images/arrow.png" width="15px"></a><a href="chap3.html">Improving the way neural networks learn</a><div id="toc_improving_the_way_neural_networks_learn" style="display: none;"><p class="toc_section"><ul><a href="chap3.html#the_cross-entropy_cost_function"><li>The cross-entropy cost function</li></a><a href="chap3.html#overfitting_and_regularization"><li>Overfitting and regularization</li></a><a href="chap3.html#weight_initialization"><li>Weight initialization</li></a><a href="chap3.html#handwriting_recognition_revisited_the_code"><li>Handwriting recognition revisited: the code</li></a><a href="chap3.html#how_to_choose_a_neural_network's_hyper-parameters"><li>How to choose a neural network's hyper-parameters?</li></a><a href="chap3.html#other_techniques"><li>Other techniques</li></a></ul></p></div>
<script>
$('#toc_improving_the_way_neural_networks_learn_reveal').click(function() { 
   var src = $('#toc_img_improving_the_way_neural_networks_learn').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_improving_the_way_neural_networks_learn").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_improving_the_way_neural_networks_learn").attr('src', 'images/arrow.png');
   };
   $('#toc_improving_the_way_neural_networks_learn').toggle('fast', function() {});  
});</script><p class='toc_mainchapter'><a id="toc_a_visual_proof_that_neural_nets_can_compute_any_function_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_a_visual_proof_that_neural_nets_can_compute_any_function" src="images/arrow.png" width="15px"></a><a href="chap4.html">A visual proof that neural nets can compute any function</a><div id="toc_a_visual_proof_that_neural_nets_can_compute_any_function" style="display: none;"><p class="toc_section"><ul><a href="chap4.html#two_caveats"><li>Two caveats</li></a><a href="chap4.html#universality_with_one_input_and_one_output"><li>Universality with one input and one output</li></a><a href="chap4.html#many_input_variables"><li>Many input variables</li></a><a href="chap4.html#extension_beyond_sigmoid_neurons"><li>Extension beyond sigmoid neurons</li></a><a href="chap4.html#fixing_up_the_step_functions"><li>Fixing up the step functions</li></a><a href="chap4.html#conclusion"><li>Conclusion</li></a></ul></p></div>
<script>
$('#toc_a_visual_proof_that_neural_nets_can_compute_any_function_reveal').click(function() { 
   var src = $('#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_a_visual_proof_that_neural_nets_can_compute_any_function").attr('src', 'images/arrow.png');
   };
   $('#toc_a_visual_proof_that_neural_nets_can_compute_any_function').toggle('fast', function() {});  
});</script><p class='toc_mainchapter'><a id="toc_why_are_deep_neural_networks_hard_to_train_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_why_are_deep_neural_networks_hard_to_train" src="images/arrow.png" width="15px"></a><a href="chap5.html">Why are deep neural networks hard to train?</a><div id="toc_why_are_deep_neural_networks_hard_to_train" style="display: none;"><p class="toc_section"><ul><a href="chap5.html#the_vanishing_gradient_problem"><li>The vanishing gradient problem</li></a><a href="chap5.html#what's_causing_the_vanishing_gradient_problem_unstable_gradients_in_deep_neural_nets"><li>What's causing the vanishing gradient problem?  Unstable gradients in deep neural nets</li></a><a href="chap5.html#unstable_gradients_in_more_complex_networks"><li>Unstable gradients in more complex networks</li></a><a href="chap5.html#other_obstacles_to_deep_learning"><li>Other obstacles to deep learning</li></a></ul></p></div>
<script>
$('#toc_why_are_deep_neural_networks_hard_to_train_reveal').click(function() { 
   var src = $('#toc_img_why_are_deep_neural_networks_hard_to_train').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_why_are_deep_neural_networks_hard_to_train").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_why_are_deep_neural_networks_hard_to_train").attr('src', 'images/arrow.png');
   };
   $('#toc_why_are_deep_neural_networks_hard_to_train').toggle('fast', function() {});  
});</script><p class='toc_mainchapter'><a id="toc_deep_learning_reveal" class="toc_reveal" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';"><img id="toc_img_deep_learning" src="images/arrow.png" width="15px"></a><a href="chap6.html">Deep learning</a><div id="toc_deep_learning" style="display: none;"><p class="toc_section"><ul><a href="chap6.html#introducing_convolutional_networks"><li>Introducing convolutional networks</li></a><a href="chap6.html#convolutional_neural_networks_in_practice"><li>Convolutional neural networks in practice</li></a><a href="chap6.html#the_code_for_our_convolutional_networks"><li>The code for our convolutional networks</li></a><a href="chap6.html#recent_progress_in_image_recognition"><li>Recent progress in image recognition</li></a><a href="chap6.html#other_approaches_to_deep_neural_nets"><li>Other approaches to deep neural nets</li></a><a href="chap6.html#on_the_future_of_neural_networks"><li>On the future of neural networks</li></a></ul></p></div>
<script>
$('#toc_deep_learning_reveal').click(function() { 
   var src = $('#toc_img_deep_learning').attr('src');
   if(src == 'images/arrow.png') {
     $("#toc_img_deep_learning").attr('src', 'images/arrow_down.png');
   } else {
     $("#toc_img_deep_learning").attr('src', 'images/arrow.png');
   };
   $('#toc_deep_learning').toggle('fast', function() {});  
});</script><p class="toc_not_mainchapter"><a href="sai.html">Appendix: Is there a <em>simple</em> algorithm for intelligence?</a></p><p class="toc_not_mainchapter"><a href="acknowledgements.html">Acknowledgements</a></p><p class="toc_not_mainchapter"><a href="faq.html">Frequently Asked Questions</a></p>
<hr>
<p class="sidebar"> If you benefit from the book, please make a small
donation.  I suggest $5, but you can choose the amount.</p>

<form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top">
<input type="hidden" name="cmd" value="_s-xclick">
<input type="hidden" name="hosted_button_id" value="5K9YAHR4X84RN">
<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
<img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1">
</form>

<p class="sidebar">Alternately, you can make a donation by sending me
Bitcoin, at address <span style="font-size: 0.7em">1Kd6tXH5SDAmiFb49J9hknG5pqj7KStSAx</span></p>

<!--
<hr>

<p class="sidebar"> If you benefit from the book, please make a small
donation.  I suggest $3, but you can choose the amount.</p>

<form action="https://www.paypal.com/cgi-bin/webscr" method="post" target="_top">
<input type="hidden" name="cmd" value="_s-xclick">
<input type="hidden" name="encrypted" value="-----BEGIN PKCS7-----MIIHTwYJKoZIhvcNAQcEoIIHQDCCBzwCAQExggEwMIIBLAIBADCBlDCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20CAQAwDQYJKoZIhvcNAQEBBQAEgYAtusFIFTgWVpgZsMgI9zMrWRAFFKQqeFiE6ay1nbmP360YzPtR+vvCXwn214Az9+F9g7mFxe0L+m9zOCdjzgRROZdTu1oIuS78i0TTbcbD/Vs/U/f9xcmwsdX9KYlhimfsya0ydPQ2xvr4iSGbwfNemIPVRCTadp/Y4OQWWRFKGTELMAkGBSsOAwIaBQAwgcwGCSqGSIb3DQEHATAUBggqhkiG9w0DBwQIK5obVTaqzmyAgajgc4w5t7l6DjTGVI7k+4UyO3uafxPac23jOyBGmxSnVRPONB9I+/Q6OqpXZtn8JpTuzFmuIgkNUf1nldv/DA1mhPOeeVxeuSGL8KpWxpJboKZ0mEu9b+0FJXvZW+snv0jodnRDtI4g0AXDZNPyRWIdJ3m+tlYfsXu4mQAe0q+CyT+QrSRhPGI/llicF4x3rMbRBNqlDze/tFqp/jbgW84Puzz6KyxAez6gggOHMIIDgzCCAuygAwIBAgIBADANBgkqhkiG9w0BAQUFADCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20wHhcNMDQwMjEzMTAxMzE1WhcNMzUwMjEzMTAxMzE1WjCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMFHTt38RMxLXJyO2SmS+Ndl72T7oKJ4u4uw+6awntALWh03PewmIJuzbALScsTS4sZoS1fKciBGoh11gIfHzylvkdNe/hJl66/RGqrj5rFb08sAABNTzDTiqqNpJeBsYs/c2aiGozptX2RlnBktH+SUNpAajW724Nv2Wvhif6sFAgMBAAGjge4wgeswHQYDVR0OBBYEFJaffLvGbxe9WT9S1wob7BDWZJRrMIG7BgNVHSMEgbMwgbCAFJaffLvGbxe9WT9S1wob7BDWZJRroYGUpIGRMIGOMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxFDASBgNVBAoTC1BheVBhbCBJbmMuMRMwEQYDVQQLFApsaXZlX2NlcnRzMREwDwYDVQQDFAhsaXZlX2FwaTEcMBoGCSqGSIb3DQEJARYNcmVAcGF5cGFsLmNvbYIBADAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GBAIFfOlaagFrl71+jq6OKidbWFSE+Q4FqROvdgIONth+8kSK//Y/4ihuE4Ymvzn5ceE3S/iBSQQMjyvb+s2TWbQYDwcp129OPIbD9epdr4tJOUNiSojw7BHwYRiPh58S1xGlFgHFXwrEBb3dgNbMUa+u4qectsMAXpVHnD9wIyfmHMYIBmjCCAZYCAQEwgZQwgY4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEUMBIGA1UEChMLUGF5UGFsIEluYy4xEzARBgNVBAsUCmxpdmVfY2VydHMxETAPBgNVBAMUCGxpdmVfYXBpMRwwGgYJKoZIhvcNAQkBFg1yZUBwYXlwYWwuY29tAgEAMAkGBSsOAwIaBQCgXTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTA4MDUxMzMyMTRaMCMGCSqGSIb3DQEJBDEWBBRtGLYvbZ45sWVegWVP2CuXTHPmJTANBgkqhkiG9w0BAQEFAASBgKgrMHMINfV7yVuZgcTjp8gUzejPF2x2zRPU/G8pKUvYIl1F38TjV2pe4w0QXcGMJRT8mQfxHCy9UmF3LfblH8F0NSMMDrZqu3M0eLk96old+L0Xl6ING8l3idFDkLagE+lZK4A0rNV35aMci3VLvjQ34CvEj7jaHeLpbkgk/l6v-----END PKCS7-----
">
<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
<img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1">
</form>

-->

<hr>
<span class="sidebar_title">Sponsors</span>
<br/>

<a href="https://lambdalabs.com/?utm_source=neuralnetworksdeeplearning&utm_medium=banner&utm_campaign=blogin&utm_content=rbannerimg">
  <img src="assets/lambda.png" width="200px" style="padding: 3px 0px 0px 10px; border-style: none;">
</a>
<br>
<div style="line-height: 1.2; padding-bottom: 12px; font-size: 0.8;">
  <a href="https://lambdalabs.com/?utm_source=neuralnetworksdeeplearning&utm_medium=banner&utm_campaign=blogin&utm_content=rtext">Deep Learning Workstations, Servers, and Laptops</a>
</div>

<a href='http://gsquaredcapital.com/'><img src='assets/gsquared.png' width='200px' style="padding: 5px 0px 10px 10px; border-style: none;"></a>

<a href='http://www.tineye.com'><img src='assets/tineye.png' width='200px'
style="padding: 0px 0px 10px 8px; border-style: none;"></a>

<a href='http://www.visionsmarts.com'><img
src='assets/visionsmarts.png' width='210px' style="padding: 0px 0px
0px 0px; border-style: none;"></a> <br/> 

<p class="sidebar">Thanks to all the <a
href="supporters.html">supporters</a> who made the book possible, with
especial thanks to Pavel Dudrenov.  Thanks also to all the
contributors to the <a href="bugfinder.html">Bugfinder Hall of
Fame</a>.  </p>

<hr>
<span class="sidebar_title">Resources</span>

<p class="sidebar"><a href="https://twitter.com/michael_nielsen">Michael Nielsen on Twitter</a></p>

<p class="sidebar"><a href="faq.html">Book FAQ</a></p>

<p class="sidebar">
<a href="https://github.com/mnielsen/neural-networks-and-deep-learning">Code repository</a></p>

<p class="sidebar">
<a href="https://michaelnielsenupdates.substack.com/subscribe">Michael Nielsen's project announcement mailing list</a>
</p>

<p class="sidebar"> <a href="http://www.deeplearningbook.org/">Deep Learning</a>, book by Ian
Goodfellow, Yoshua Bengio, and Aaron Courville</p>

<p class="sidebar"><a href="http://cognitivemedium.com">cognitivemedium.com</a></p>

<hr>
<a href="http://michaelnielsen.org"><img src="assets/Michael_Nielsen_Web_Small.jpg" width="160px" style="border-style: none;"/></a>

<p class="sidebar">
By <a href="http://michaelnielsen.org">Michael Nielsen</a> / Dec 2019
</p>
</div>
</p><p>In the <a href="chap5.html">last chapter</a> we learned that deep neuralnetworks are often much harder to train than shallow neural networks.That's unfortunate, since we have good reason to believe that<em>if</em> we could train deep nets they'd be much more powerful thanshallow nets.  But while the news from the last chapter isdiscouraging, we won't let it stop us.  In this chapter, we'll developtechniques which can be used to train deep networks, and apply them inpractice.  We'll also look at the broader picture, briefly reviewingrecent progress on using deep nets for image recognition, speechrecognition, and other applications.  And we'll take a brief,speculative look at what the future may hold for neural nets, and forartificial intelligence.</p><p>The chapter is a long one.  To help you navigate, let's take a tour.The sections are only loosely coupled, so provided you have some basicfamiliarity with neural nets, you can jump to whatever most interestsyou.</p><p>The <a href="chap6.html#convolutional_networks">main part of the chapter</a> is anintroduction to one of the most widely used types of deep network:deep convolutional networks.  We'll work through a detailed example- code and all - of using convolutional nets to solve the problemof classifying handwritten digits from the MNIST data set:</p><p><center><img src="images/digits.png" width="160px"></center></p><p>We'll start our account of convolutional networks with the shallownetworks used to attack this problem earlier in the book.  Throughmany iterations we'll build up more and more powerful networks.  As wego we'll explore many powerful techniques: convolutions, pooling, theuse of GPUs to do far more training than we did with our shallownetworks, the algorithmic expansion of our training data (to reduceoverfitting), the use of the dropout technique (also to reduceoverfitting), the use of ensembles of networks, and others.  Theresult will be a system that offers near-human performance.  Of the10,000 MNIST test images - images not seen during training! - oursystem will classify 9,967 correctly.  Here's a peek at the 33 imageswhich are misclassified.  Note that the correct classification is inthe top right; our program's classification is in the bottom right:</p><p><center><img src="images/ensemble_errors.png" width="580px"></center></p><p>Many of these are tough even for a human to classify.  Consider, forexample, the third image in the top row.  To me it looks more like a"9" than an "8", which is the official classification.  Ournetwork also thinks it's a "9".  This kind of "error" is at thevery least understandable, and perhaps even commendable.  We concludeour discussion of image recognition with a<a href="chap6.html#recent_progress_in_image_recognition">survey of some of the  spectacular recent progress</a> using networks (particularlyconvolutional nets) to do image recognition.</p><p>The remainder of the chapter discusses deep learning from a broaderand less detailed perspective.  We'll<a href="chap6.html#things_we_didn't_cover_but_which_you'll_eventually_want_to_know">briefly  survey other models of neural networks</a>, such as recurrent neuralnets and long short-term memory units, and how such models can beapplied to problems in speech recognition, natural languageprocessing, and other areas.  And we'll<a href="chap6.html#on_the_future_of_neural_networks">speculate about the  future of neural networks and deep learning</a>, ranging from ideaslike intention-driven user interfaces, to the role of deep learning inartificial intelligence.</p><p></p><p></p><p></p><p></p><p></p><p></p><p>The chapter builds on the earlier chapters in the book, making use ofand integrating ideas such as backpropagation, regularization, thesoftmax function, and so on.  However, to read the chapter you don'tneed to have worked in detail through all the earlier chapters.  Itwill, however, help to have read <a href="chap1.html">Chapter 1</a>, on thebasics of neural networks.  When I use concepts from Chapters 2 to 5,I provide links so you can familiarize yourself, if necessary.</p><p>It's worth noting what the chapter is not.  It's not a tutorial on thelatest and greatest neural networks libraries.  Nor are we going to betraining deep networks with dozens of layers to solve problems at thevery leading edge.  Rather, the focus is on understanding some of thecore principles behind deep neural networks, and applying them in thesimple, easy-to-understand context of the MNIST problem.  Put anotherway: the chapter is not going to bring you right up to the frontier.Rather, the intent of this and earlier chapters is to focus onfundamentals, and so to prepare you to understand a wide range ofcurrent work.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p><h3><a name="introducing_convolutional_networks"></a><a href="chap6.html#introducing_convolutional_networks">Introducing convolutional networks</a></h3></p><p>In earlier chapters, we taught our neural networks to do a pretty goodjob recognizing images of handwritten digits:</p><p><center><img src="images/digits.png" width="160px"></center></p><p>We did this using networks in which adjacent network layers are fullyconnected to one another.  That is, every neuron in the network isconnected to every neuron in adjacent layers:</p><p><center><img src="images/tikz41.png"/></center></p><p>In particular, for each pixel in the input image, we encoded thepixel's intensity as the value for a corresponding neuron in the inputlayer.  For the $28 \times 28$ pixel images we've been using, thismeans our network has $784$ ($= 28 \times 28$) input neurons.  We thentrained the network's weights and biases so that the network's outputwould - we hope! - correctly identify the input image: '0', '1','2', ..., '8', or '9'.</p><p>Our earlier networks work pretty well: we've<a href="chap3.html#98percent">obtained a classification accuracy better  than 98 percent</a>, using training and test data from the<a href="chap1.html#learning_with_gradient_descent">MNIST handwritten  digit data set</a>.  But upon reflection, it's strange to use networkswith fully-connected layers to classify images.  The reason is thatsuch a network architecture does not take into account the spatialstructure of the images.  For instance, it treats input pixels whichare far apart and close together on exactly the same footing.  Suchconcepts of spatial structure must instead be inferred from thetraining data.  But what if, instead of starting with a networkarchitecture which is <em>tabula rasa</em>, we used an architecturewhich tries to take advantage of the spatial structure?  In thissection I describe <em>convolutional neural networks</em>*<span class="marginnote">*The  origins of convolutional neural networks go back to the 1970s.  But  the seminal paper establishing the modern subject of convolutional  networks was a 1998 paper,  <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf">"Gradient-based    learning applied to document recognition"</a>, by Yann LeCun,  Léon Bottou, Yoshua Bengio, and Patrick Haffner.  LeCun has since made an interesting  <a href="https://www.facebook.com/yann.lecun/posts/10152348155137143">remark</a>  on the terminology for convolutional nets: "The [biological] neural  inspiration in models like convolutional nets is very  tenuous. That's why I call them 'convolutional nets' not  'convolutional neural nets', and why we call the nodes 'units' and  not 'neurons' ".  Despite this remark, convolutional nets use many  of the same ideas as the neural networks we've studied up to now:  ideas such as backpropagation, gradient descent, regularization,  non-linear activation functions, and so on.  And so we will follow  common practice, and consider them a type of neural network.  I will  use the terms "convolutional neural network" and "convolutional  net(work)" interchangeably.  I will also use the terms  "[artificial] neuron" and "unit" interchangeably.</span>.  Thesenetworks use a special architecture which is particularly well-adaptedto classify images.  Using this architecture makes convolutionalnetworks fast to train.  This, in turn, helps us train deep,many-layer networks, which are very good at classifying images.Today, deep convolutional networks or some close variant are used inmost neural networks for image recognition.</p><p>Convolutional neural networks use three basic ideas: <em>local  receptive fields</em>, <em>shared weights</em>, and <em>pooling</em>.  Let'slook at each of these ideas in turn.</p><p><strong>Local receptive fields:</strong> In the fully-connected layers shownearlier, the inputs were depicted as a vertical line of neurons.  In aconvolutional net, it'll help to think instead of the inputs as a $28\times 28$ square of neurons, whose values correspond to the $28\times 28$ pixel intensities we're using as inputs:</p><p><center><img src="images/tikz42.png"/></center></p><p>As per usual, we'll connect the input pixels to a layer of hiddenneurons.  But we won't connect every input pixel to every hiddenneuron.  Instead, we only make connections in small, localized regionsof the input image.</p><p>To be more precise, each neuron in the first hidden layer will beconnected to a small region of the input neurons, say, for example, a$5 \times 5$ region, corresponding to $25$ input pixels.  So, for aparticular hidden neuron, we might have connections that look likethis:<center><img src="images/tikz43.png"/></center></p><p>That region in the input image is called the <em>local receptive  field</em> for the hidden neuron.  It's a little window on the inputpixels.  Each connection learns a weight.  And the hidden neuronlearns an overall bias as well.  You can think of that particularhidden neuron as learning to analyze its particular local receptivefield.</p><p>We then slide the local receptive field across the entire input image.For each local receptive field, there is a different hidden neuron inthe first hidden layer.  To illustrate this concretely, let's startwith a local receptive field in the top-left corner:<center><img src="images/tikz44.png"/></center></p><p>Then we slide the local receptive field over by one pixel to the right(i.e., by one neuron), to connect to a second hidden neuron:</p><p><center><img src="images/tikz45.png"/></center></p><p>And so on, building up the first hidden layer.  Note that if we have a$28 \times 28$ input image, and $5 \times 5$ local receptive fields,then there will be $24 \times 24$ neurons in the hidden layer.  Thisis because we can only move the local receptive field $23$ neuronsacross (or $23$ neurons down), before colliding with the right-handside (or bottom) of the input image.</p><p>I've shown the local receptive field being moved by one pixel at atime.  In fact, sometimes a different <em>stride length</em> is used.For instance, we might move the local receptive field $2$ pixels tothe right (or down), in which case we'd say a stride length of $2$ isused.  In this chapter we'll mostly stick with stride length $1$, butit's worth knowing that people sometimes experiment with differentstride lengths*<span class="marginnote">*As was done in earlier chapters, if we're  interested in trying different stride lengths then we can use  validation data to pick out the stride length which gives the best  performance.  For more details, see the  <a href="chap3.html#how_to_choose_a_neural_network's_hyper-parameters">earlier    discussion</a> of how to choose hyper-parameters in a neural network.  The same approach may also be used to choose the size of the local  receptive field - there is, of course, nothing special about using  a $5 \times 5$ local receptive field.  In general, larger local  receptive fields tend to be helpful when the input images are  significantly larger than the $28 \times 28$ pixel MNIST images.</span>.</p><p><strong>Shared weights and biases:</strong> I've said that each hidden neuronhas a bias and $5 \times 5$ weights connected to its local receptivefield.  What I did not yet mention is that we're going to use the<em>same</em> weights and bias for each of the $24 \times 24$ hiddenneurons.  In other words, for the $j, k$th hidden neuron, the outputis:<a class="displaced_anchor" name="eqtn125"></a>\begin{eqnarray}   \sigma\left(b + \sum_{l=0}^4 \sum_{m=0}^4  w_{l,m} a_{j+l, k+m} \right).\tag{125}\end{eqnarray}Here, $\sigma$ is the neural activation function - perhaps the<a href="chap1.html#sigmoid_neurons">sigmoid function</a> we used inearlier chapters.  $b$ is the shared value for the bias.  $w_{l,m}$ isa $5 \times 5$ array of shared weights.  And, finally, we use $a_{x,  y}$ to denote the input activation at position $x, y$.</p><p>This means that all the neurons in the first hidden layer detectexactly the same feature*<span class="marginnote">*I haven't precisely defined the  notion of a feature.  Informally, think of the feature detected by a  hidden neuron as the kind of input pattern that will cause the  neuron to activate: it might be an edge in the image, for instance,  or maybe some other type of shape. </span>, just at different locations inthe input image.  To see why this makes sense, suppose the weights andbias are such that the hidden neuron can pick out, say, a verticaledge in a particular local receptive field.  That ability is alsolikely to be useful at other places in the image.  And so it is usefulto apply the same feature detector everywhere in the image.  To put itin slightly more abstract terms, convolutional networks are welladapted to the translation invariance of images: move a picture of acat (say) a little ways, and it's still an image of a cat*<span class="marginnote">*In  fact, for the MNIST digit classification problem we've been  studying, the images are centered and size-normalized.  So MNIST has  less translation invariance than images found "in the wild", so to  speak.  Still, features like edges and corners are likely to be  useful across much of the input space. </span>.</p><p>For this reason, we sometimes call the map from the input layer to thehidden layer a <em>feature map</em>.  We call the weights defining thefeature map the <em>shared weights</em>.  And we call the bias definingthe feature map in this way the <em>shared bias</em>.  The sharedweights and bias are often said to define a <em>kernel</em> or<em>filter</em>.  In the literature, people sometimes use these terms inslightly different ways, and for that reason I'm not going to be moreprecise; rather, in a moment, we'll look at some concrete examples.</p><p></p><p>The network structure I've described so far can detect just a singlekind of localized feature.  To do image recognition we'll need morethan one feature map.  And so a complete convolutional layer consistsof several different feature maps:</p><p><center><img src="images/tikz46.png"/></center>In the example shown, there are $3$ feature maps.  Each feature map isdefined by a set of $5 \times 5$ shared weights, and a single sharedbias.  The result is that the network can detect $3$ different kindsof features, with each feature being detectable across the entireimage.</p><p></p><p>I've shown just $3$ feature maps, to keep the diagram above simple.However, in practice convolutional networks may use more (and perhapsmany more) feature maps.  One of the early convolutional networks,LeNet-5, used $6$ feature maps, each associated to a $5 \times 5$local receptive field, to recognize MNIST digits.  So the exampleillustrated above is actually pretty close to LeNet-5.  In theexamples we develop later in the chapter we'll use convolutionallayers with $20$ and $40$ feature maps.  Let's take a quick peek atsome of the features which are learned*<span class="marginnote">*The feature maps  illustrated come from the final convolutional network we train, see  <a href="chap6.html#final_conv">here</a>.</span>:</p><p><center><img src="images/net_full_layer_0.png" width="400px"></center></p><p>The $20$ images correspond to $20$ different feature maps (or filters,or kernels).  Each map is represented as a $5 \times 5$ block image,corresponding to the $5 \times 5$ weights in the local receptivefield.  Whiter blocks mean a smaller (typically, more negative)weight, so the feature map responds less to corresponding inputpixels.  Darker blocks mean a larger weight, so the feature mapresponds more to the corresponding input pixels.  Very roughlyspeaking, the images above show the type of features the convolutionallayer responds to.</p><p>So what can we conclude from these feature maps?  It's clear there isspatial structure here beyond what we'd expect at random: many of thefeatures have clear sub-regions of light and dark.  That shows ournetwork really is learning things related to the spatial structure.However, beyond that, it's difficult to see what these featuredetectors are learning.  Certainly, we're not learning (say) the<a href="http://en.wikipedia.org/wiki/Gabor_filter">Gabor filters</a> whichhave been used in many traditional approaches to image recognition.In fact, there's now a lot of work on better understanding thefeatures learnt by convolutional networks.  If you're interested infollowing up on that work, I suggest starting with the paper<a href="http://arxiv.org/abs/1311.2901">Visualizing and Understanding  Convolutional Networks</a> by Matthew Zeiler and Rob Fergus (2013).</p><p></p><p>A big advantage of sharing weights and biases is that it greatlyreduces the number of parameters involved in a convolutional network.For each feature map we need $25 = 5 \times 5$ shared weights, plus asingle shared bias. So each feature map requires $26$ parameters.  Ifwe have $20$ feature maps that's a total of $20 \times 26 = 520$parameters defining the convolutional layer.  By comparison, supposewe had a fully connected first layer, with $784 = 28 \times 28$ inputneurons, and a relatively modest $30$ hidden neurons, as we used inmany of the examples earlier in the book.  That's a total of $784\times 30$ weights, plus an extra $30$ biases, for a total of $23,550$parameters.  In other words, the fully-connected layer would have morethan $40$ times as many parameters as the convolutional layer.</p><p>Of course, we can't really do a direct comparison between the numberof parameters, since the two models are different in essential ways.But, intuitively, it seems likely that the use of translationinvariance by the convolutional layer will reduce the number ofparameters it needs to get the same performance as the fully-connectedmodel.  That, in turn, will result in faster training for theconvolutional model, and, ultimately, will help us build deep networksusing convolutional layers.</p><p></p><p>Incidentally, the name <em>convolutional</em> comes from the fact thatthe operation in Equation <span id="margin_736057893021_reveal" class="equation_link">(125)</span><span id="margin_736057893021" class="marginequation" style="display: none;"><a href="chap6.html#eqtn125" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray}   \sigma\left(b + \sum_{l=0}^4 \sum_{m=0}^4  w_{l,m} a_{j+l, k+m} \right) \nonumber\end{eqnarray}</a></span><script>$('#margin_736057893021_reveal').click(function() {$('#margin_736057893021').toggle('slow', function() {});});</script> is sometimes known as a<em>convolution</em>.  A little more precisely, people sometimes writethat equation as $a^1 = \sigma(b + w * a^0)$, where $a^1$ denotes theset of output activations from one feature map, $a^0$ is the set ofinput activations, and $*$ is called a convolution operation.  We'renot going to make any deep use of the mathematics of convolutions, soyou don't need to worry too much about this connection.  But it'sworth at least knowing where the name comes from.</p><p></p><p></p><p></p><p></p><p><strong>Pooling layers:</strong> In addition to the convolutional layers justdescribed, convolutional neural networks also contain <em>pooling  layers</em>.  Pooling layers are usually used immediately afterconvolutional layers.  What the pooling layers do is simplify theinformation in the output from the convolutional layer.</p><p></p><p>In detail, a pooling layer takes each feature map*<span class="marginnote">*The  nomenclature is being used loosely here.  In particular, I'm using  "feature map" to mean not the function computed by the  convolutional layer, but rather the activation of the hidden neurons  output from the layer.  This kind of mild abuse of nomenclature is  pretty common in the research literature.</span> output from theconvolutional layer and prepares a condensed feature map.  Forinstance, each unit in the pooling layer may summarize a region of(say) $2 \times 2$ neurons in the previous layer.  As a concreteexample, one common procedure for pooling is known as<em>max-pooling</em>.  In max-pooling, a pooling unit simply outputs themaximum activation in the $2 \times 2$ input region, as illustrated inthe following diagram:</p><p><center><img src="images/tikz47.png"/></center></p><p>Note that since we have $24 \times 24$ neurons output from theconvolutional layer, after pooling we have $12 \times 12$ neurons.</p><p>As mentioned above, the convolutional layer usually involves more thana  single feature  map.   We  apply max-pooling  to  each feature  mapseparately.   So  if  there  were three  feature  maps,  the  combinedconvolutional and max-pooling layers would look like:</p><p><center><img src="images/tikz48.png"/></center></p><p>We can think of max-pooling as a way for the network to ask whether agiven feature is found anywhere in a region of the image.  It thenthrows away the exact positional information.  The intuition is thatonce a feature has been found, its exact location isn't as importantas its rough location relative to other features.  A big benefit isthat there are many fewer pooled features, and so this helps reducethe number of parameters needed in later layers.</p><p></p><p>Max-pooling isn't the only technique used for pooling.  Another commonapproach is known as <em>L2 pooling</em>.  Here, instead of taking themaximum activation of a $2 \times 2$ region of neurons, we take thesquare root of the sum of the squares of the activations in the $2\times 2$ region.  While the details are different, the intuition issimilar to max-pooling: L2 pooling is a way of condensing informationfrom the convolutional layer.  In practice, both techniques have beenwidely used.  And sometimes people use other types of poolingoperation.  If you're really trying to optimize performance, you mayuse validation data to compare several different approaches topooling, and choose the approach which works best.  But we're notgoing to worry about that kind of detailed optimization.</p><p></p><p><strong>Putting it all together:</strong> We can now put all these ideastogether to form a complete convolutional neural network.  It'ssimilar to the architecture we were just looking at, but has theaddition of a layer of $10$ output neurons, corresponding to the $10$possible values for MNIST digits ('0', '1', '2', <em>etc</em>):</p><p><center><img src="images/tikz49.png"/></center></p><p>The network begins with $28 \times 28$ input neurons, which are usedto encode the pixel intensities for the MNIST image.  This is thenfollowed by a convolutional layer using a $5 \times 5$ local receptivefield and $3$ feature maps.  The result is a layer of $3 \times 24\times 24$ hidden feature neurons.  The next step is a max-poolinglayer, applied to $2 \times 2$ regions, across each of the $3$ featuremaps.  The result is a layer of $3 \times 12 \times 12$ hidden featureneurons.</p><p>The final layer of connections in the network is a fully-connectedlayer.  That is, this layer connects <em>every</em> neuron from themax-pooled layer to every one of the $10$ output neurons.  Thisfully-connected architecture is the same as we used in earlierchapters.  Note, however, that in the diagram above, I've used asingle arrow, for simplicity, rather than showing all the connections.Of course, you can easily imagine the connections.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p>This convolutional architecture is quite different to thearchitectures used in earlier chapters.  But the overall picture issimilar: a network made of many simple units, whose behaviors aredetermined by their weights and biases.  And the overall goal is stillthe same: to use training data to train the network's weights andbiases so that the network does a good job classifying input digits.</p><p>In particular, just as earlier in the book, we will train our networkusing stochastic gradient descent and backpropagation.  This mostlyproceeds in exactly the same way as in earlier chapters.  However, wedo need to make a few modifications to the backpropagation procedure.The reason is that our earlier <a href="chap2.html">derivation of  backpropagation</a> was for networks with fully-connected layers.Fortunately, it's straightforward to modify the derivation forconvolutional and max-pooling layers.  If you'd like to understand thedetails, then I invite you to work through the following problem.  Bewarned that the problem will take some time to work through, unlessyou've really internalized the <a href="chap2.html">earlier derivation of  backpropagation</a> (in which case it's easy).</p><p><h4><a name="problem_366128"></a><a href="chap6.html#problem_366128">Problem</a></h4><ul><li><strong>Backpropagation in a convolutional network</strong> The core equations  of backpropagation in a network with fully-connected layers  are <span id="margin_209324202488_reveal" class="equation_link">(BP1)</span><span id="margin_209324202488" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP1" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray}   \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j) \nonumber\end{eqnarray}</a></span><script>$('#margin_209324202488_reveal').click(function() {$('#margin_209324202488').toggle('slow', function() {});});</script>-<span id="margin_759010070359_reveal" class="equation_link">(BP4)</span><span id="margin_759010070359" class="marginequation" style="display: none;"><a href="chap2.html#eqtnBP4" style="padding-bottom: 5px;" onMouseOver="this.style.borderBottom='1px solid #2A6EA6';" onMouseOut="this.style.borderBottom='0px';">\begin{eqnarray}    \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j \nonumber\end{eqnarray}</a></span><script>$('#margin_759010070359_reveal').click(function() {$('#margin_759010070359').toggle('slow', function() {});});</script>  (<a href="chap2.html#backpropsummary">link</a>).  Suppose we have a  network containing a convolutional layer, a max-pooling layer, and a  fully-connected output layer, as in the network discussed above.  How are the equations of backpropagation modified?</ul></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p><h3><a name="convolutional_neural_networks_in_practice"></a><a href="chap6.html#convolutional_neural_networks_in_practice">Convolutional neural networks in practice</a></h3></p><p>We've now seen the core ideas behind convolutional neural networks.Let's look at how they work in practice, by implementing someconvolutional networks, and applying them to the MNIST digitclassification problem.  The program we'll use to do this is called<tt>network3.py</tt>, and it's an improved version of the programs<tt>network.py</tt> and <tt>network2.py</tt> developed in earlierchapters*<span class="marginnote">*Note also that <tt>network3.py</tt> incorporates ideas  from the Theano library's documentation on convolutional neural nets  (notably the implementation of  <a href="http://deeplearning.net/tutorial/lenet.html">LeNet-5</a>), from  Misha Denil's  <a href="https://github.com/mdenil/dropout">implementation of dropout</a>,  and from <a href="http://colah.github.io">Chris Olah</a>.</span>.  If you wishto follow along, the code is available<a href="https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network3.py">on  GitHub</a>.  Note that we'll work through the code for<tt>network3.py</tt> itself in the next section.  In this section, we'lluse <tt>network3.py</tt> as a library to build convolutional networks.</p><p></p><p>The programs <tt>network.py</tt> and <tt>network2.py</tt> were implementedusing Python and the matrix library Numpy.  Those programs worked fromfirst principles, and got right down into the details ofbackpropagation, stochastic gradient descent, and so on.  But now thatwe understand those details, for <tt>network3.py</tt> we're going to usea machine learning library known as<a href="http://deeplearning.net/software/theano/">Theano</a>*<span class="marginnote">*See  <a href="http://www.iro.umontreal.ca/&#126;lisa/pointeurs/theano_scipy2010.pdf">Theano:    A CPU and GPU Math Expression Compiler in Python</a>, by James  Bergstra, Olivier Breuleux, Frederic Bastien, Pascal Lamblin, Ravzan  Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley,  and Yoshua Bengio (2010).  Theano is also the basis for the popular  <a href="http://deeplearning.net/software/pylearn2/">Pylearn2</a> and  <a href="http://keras.io/">Keras</a> neural networks libraries. Other  popular neural nets libraries at the time of this writing include  <a href="http://caffe.berkeleyvision.org">Caffe</a> and  <a href="http://torch.ch">Torch</a>. </span>.  Using Theano makes it easy toimplement backpropagation for convolutional neural networks, since itautomatically computes all the mappings involved.  Theano is alsoquite a bit faster than our earlier code (which was written to be easyto understand, not fast), and this makes it practical to train morecomplex networks.  In particular, one great feature of Theano is thatit can run code on either a CPU or, if available, a GPU.  Running on aGPU provides a substantial speedup and, again, helps make it practicalto train more complex networks.</p><p></p><p>If you wish to follow along, then you'll need to get Theano running onyour system.  To install Theano, follow the instructions at theproject's <a href="http://deeplearning.net/software/theano/">homepage</a>.The examples which follow were run using Theano 0.6*<span class="marginnote">*As I  release this chapter, the current version of Theano has changed to  version 0.7.  I've actually rerun the examples under Theano 0.7 and  get extremely similar results to those reported in the text.</span>.  Somewere run under Mac OS X Yosemite, with no GPU.  Some were run onUbuntu 14.04, with an NVIDIA GPU.  And some of the experiments were rununder both.  To get <tt>network3.py</tt> running you'll need to set the<tt>GPU</tt> flag to either <tt>True</tt> or <tt>False</tt> (as appropriate)in the <tt>network3.py</tt> source.  Beyond that, to get Theano up andrunning on a GPU you may find<a href="http://deeplearning.net/software/theano/tutorial/using_gpu.html">the  instructions here</a> helpful.  There are also tutorials on the web,easily found using Google, which can help you get things working.  Ifyou don't have a GPU available locally, then you may wish to look into<a href="http://aws.amazon.com/ec2/instance-types/">Amazon Web Services</a>EC2 G2 spot instances.  Note that even with a GPU the code will takesome time to execute.  Many of the experiments take from minutes tohours to run.  On a CPU it may take days to run the most complex ofthe experiments.  As in earlier chapters, I suggest setting thingsrunning, and continuing to read, occasionally coming back to check theoutput from the code.  If you're using a CPU, you may wish to reducethe number of training epochs for the more complex experiments, orperhaps omit them entirely.</p><p>To get a baseline, we'll start with a shallow architecture using justa single hidden layer, containing $100$ hidden neurons.  We'll trainfor $60$ epochs, using a learning rate of $\eta = 0.1$, a mini-batchsize of $10$, and no regularization.  Here we go*<span class="marginnote">*Code for the  experiments in this section may be found  <a href="https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/conv.py">in    this script</a>.  Note that the code in the script simply duplicates  and parallels the discussion in this section.<br><br>Note also that  throughout the section I've explicitly specified the number of  training epochs.  I've done this for clarity about how we're  training.  In practice, it's worth using  <a href="chap3.html#early_stopping">early stopping</a>, that is,  tracking accuracy on the validation set, and stopping training when  we are confident the validation accuracy has stopped improving.</span>:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">network3</span>
<span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">network3</span> <span class="kn">import</span> <span class="n">Network</span>
<span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">network3</span> <span class="kn">import</span> <span class="n">ConvPoolLayer</span><span class="p">,</span> <span class="n">FullyConnectedLayer</span><span class="p">,</span> <span class="n">SoftmaxLayer</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">training_data</span><span class="p">,</span> <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span> <span class="o">=</span> <span class="n">network3</span><span class="o">.</span><span class="n">load_data_shared</span><span class="p">()</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">mini_batch_size</span> <span class="o">=</span> <span class="mi">10</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">784</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">)</span>
</pre></div>
</p><p></p><p>I obtained a best classification accuracy of $97.80$ percent. This isthe classification accuracy on the <tt>test_data</tt>, evaluated at thetraining epoch where we get the best classification accuracy on the<tt>validation_data</tt>. Using the validation data to decide when toevaluate the test accuracy helps avoid overfitting to the test data(see this <a href="chap3.html#validation_explanation">earlier  discussion</a> of the use of validation data).  We will follow thispractice below.  Your results may vary slightly, since the network'sweights and biases are randomly initialized*<span class="marginnote">*In fact, in this  experiment I actually did three separate runs training a network  with this architecture.  I then reported the test accuracy which  corresponded to the best validation accuracy from any of the three  runs.  Using multiple runs helps reduce variation in results, which  is useful when comparing many architectures, as we are doing.  I've  followed this procedure below, except where noted.  In practice, it  made little difference to the results obtained.</span>.</p><p>This $97.80$ percent accuracy is close to the $98.04$ percent accuracyobtained back in <a href="chap3.html#chap3_98_04_percent">Chapter 3</a>,using a similar network architecture and learning hyper-parameters.In particular, both examples used a shallow network, with a singlehidden layer containing $100$ hidden neurons.  Both also trained for$60$ epochs, used a mini-batch size of $10$, and a learning rate of$\eta = 0.1$.</p><p>There were, however, two differences in the earlier network.  First,we <a href="chap3.html#overfitting_and_regularization">regularized</a>the earlier network, to help reduce the effects ofoverfitting. Regularizing the current network does improve theaccuracies, but the gain is only small, and so we'll hold off worryingabout regularization until later.  Second, while the final layer inthe earlier network used sigmoid activations and the cross-entropycost function, the current network uses a softmax final layer, and thelog-likelihood cost function.  As<a href="chap3.html#softmax">explained</a> in Chapter 3 this isn't a bigchange.  I haven't made this switch for any particularly deep reason- mostly, I've done it because softmax plus log-likelihood cost ismore common in modern image classification networks.</p><p>Can we do better than these results using a deeper networkarchitecture?</p><p>Let's begin by inserting a convolutional layer, right at the beginningof the network.  We'll use $5$ by $5$ local receptive fields, a stridelength of $1$, and $20$ feature maps.  We'll also insert a max-poolinglayer, which combines the features using $2$ by $2$ pooling windows.So the overall network architecture looks much like the architecturediscussed in the last section, but with an extra fully-connectedlayer:</p><p><center><img src="images/simple_conv.png" width="550px"></center></p><p>In this architecture, we can think of the convolutional and poolinglayers as learning about local spatial structure in the input trainingimage, while the later, fully-connected layer learns at a moreabstract level, integrating global information from across the entireimage.  This is a common pattern in convolutional neural networks.</p><p>Let's train such a network, and see how it performs*<span class="marginnote">*I've  continued to use a mini-batch size of $10$ here.  In fact, as we  <a href="chap3.html#mini_batch_size">discussed earlier</a> it may be  possible to speed up training using larger mini-batches.  I've  continued to use the same mini-batch size mostly for consistency  with the experiments in earlier chapters.</span>:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">20</span><span class="o">*</span><span class="mi">12</span><span class="o">*</span><span class="mi">12</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">)</span>   
</pre></div>
</p><p></p><p></p><p>That gets us to $98.78$ percent accuracy, which is a considerableimprovement over any of our previous results.  Indeed, we've reducedour error rate by better than a third, which is a great improvement.</p><p>In specifying the network structure, I've treated the convolutionaland pooling layers as a single layer.  Whether they're regarded asseparate layers or as a single layer is to some extent a matter oftaste.  <tt>network3.py</tt> treats them as a single layer because itmakes the code for <tt>network3.py</tt> a little more compact.  However,it is easy to modify <tt>network3.py</tt> so the layers can be specifiedseparately, if desired.</p><p><h4><a name="exercise_683491"></a><a href="chap6.html#exercise_683491">Exercise</a></h4><ul><li> What classification accuracy do you get if you omit the  fully-connected layer, and just use the convolutional-pooling layer  and softmax layer?  Does the inclusion of the fully-connected layer  help?             </ul></p><p>Can we improve on the $98.78$ percent classification accuracy?</p><p>Let's try inserting a second convolutional-pooling layer.  We'll makethe insertion between the existing convolutional-pooling layer and thefully-connected hidden layer.  Again, we'll use a $5 \times 5$ localreceptive field, and pool over $2 \times 2$ regions.  Let's see whathappens when we train using similar hyper-parameters to before:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">12</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">40</span><span class="o">*</span><span class="mi">4</span><span class="o">*</span><span class="mi">4</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">)</span>        
</pre></div>
</p><p></p><p></p><p>Once again, we get an improvement: we're now at $99.06$ percentclassification accuracy!</p><p>There's two natural questions to ask at this point.  The firstquestion is: what does it even mean to apply a secondconvolutional-pooling layer?  In fact, you can think of the secondconvolutional-pooling layer as having as input $12 \times 12$"images", whose "pixels" represent the presence (or absence) ofparticular localized features in the original input image.  So you canthink of this layer as having as input a version of the original inputimage.  That version is abstracted and condensed, but still has a lotof spatial structure, and so it makes sense to use a secondconvolutional-pooling layer.</p><p>That's a satisfying point of view, but gives rise to a secondquestion.  The output from the previous layer involves $20$ separatefeature maps, and so there are $20 \times 12 \times 12$ inputs to thesecond convolutional-pooling layer.  It's as though we've got $20$separate images input to the convolutional-pooling layer, not a singleimage, as was the case for the first convolutional-pooling layer.  Howshould neurons in the second convolutional-pooling layer respond tothese multiple input images?  In fact, we'll allow each neuron in thislayer to learn from <em>all</em> $20 \times 5 \times 5$ input neurons inits local receptive field.  More informally: the feature detectors inthe second convolutional-pooling layer have access to <em>all</em> thefeatures from the previous layer, but only within their particularlocal receptive field*<span class="marginnote">*This issue would have arisen in the  first layer if the input images were in color.  In that case we'd  have 3 input features for each pixel, corresponding to red, green  and blue channels in the input image.  So we'd allow the feature  detectors to have access to all color information, but only within a  given local receptive field.</span>.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p><h4><a name="problem_834310"></a><a href="chap6.html#problem_834310">Problem</a></h4><ul><li><strong>Using the tanh activation function</strong> Several times earlier in the  book I've mentioned arguments that the  <a href="chap3.html#other_models_of_artificial_neuron">tanh    function</a> may be a better activation function than the sigmoid  function.  We've never acted on those suggestions, since we were  already making plenty of progress with the sigmoid.  But now let's  try some experiments with tanh as our activation function.  Try  training the network with tanh activations in the convolutional and  fully-connected layers*<span class="marginnote">*Note that you can pass    <tt>activation_fn=tanh</tt> as a parameter to the    <tt>ConvPoolLayer</tt> and <tt>FullyConnectedLayer</tt> classes.</span>.  Begin with the same hyper-parameters as for the sigmoid network, but  train for $20$ epochs instead of $60$.  How well does your network  perform?  What if you continue out to $60$ epochs?  Try plotting the  per-epoch validation accuracies for both tanh- and sigmoid-based  networks, all the way out to $60$ epochs.  If your results are  similar to mine, you'll find the tanh networks train a little  faster, but the final accuracies are very similar.  Can you explain  why the tanh network might train faster?  Can you get a similar  training speed with the sigmoid, perhaps by changing the learning  rate, or doing some rescaling*<span class="marginnote">*You may perhaps find    inspiration in recalling that $\sigma(z) = (1+\tanh(z/2))/2$.</span>?  Try a half-dozen iterations on the learning hyper-parameters or  network architecture, searching for ways that tanh may be superior  to the sigmoid. <em>Note: This is an open-ended problem.    Personally, I did not find much advantage in switching to tanh,    although I haven't experimented exhaustively, and perhaps you may    find a way.  In any case, in a moment we will find an advantage in    switching to the rectified linear activation function, and so we    won't go any deeper into the use of tanh.</em></ul></p><p></p><p><strong>Using rectified linear units:</strong> The network we've developed atthis point is actually a variant of one of the networks used in theseminal 1998paper*<span class="marginnote">*<a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf">"Gradient-based    learning applied to document recognition"</a>, by Yann LeCun,  Léon Bottou, Yoshua Bengio, and Patrick Haffner  (1998).  There are many differences of detail, but broadly speaking  our network is quite similar to the networks described in the  paper.</span> introducing the MNIST problem, a network known as LeNet-5.It's a good foundation for further experimentation, and for buildingup understanding and intuition.  In particular, there are many ways wecan vary the network in an attempt to improve our results.</p><p>As a beginning, let's change our neurons so that instead of using asigmoid activation function, we use<a href="chap3.html#other_models_of_artificial_neuron">rectified  linear units</a>.  That is, we'll use the activation function $f(z)\equiv \max(0, z)$.  We'll train for $60$ epochs, with a learning rateof $\eta = 0.03$.  I also found that it helps a little to use some<a href="chap3.html#overfitting_and_regularization">l2  regularization</a>, with regularization parameter $\lambda = 0.1$:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">network3</span> <span class="kn">import</span> <span class="n">ReLU</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">12</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">40</span><span class="o">*</span><span class="mi">4</span><span class="o">*</span><span class="mi">4</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.03</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="n">lmbda</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span>
</pre></div>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p>I obtained a classification accuracy of $99.23$ percent.  It's amodest improvement over the sigmoid results ($99.06$).  However,across all my experiments I found that networks based on rectifiedlinear units consistently outperformed networks based on sigmoidactivation functions.  There appears to be a real gain in moving torectified linear units for this problem.</p><p>What makes the rectified linear activation function better than thesigmoid or tanh functions?  At present, we have a poor understandingof the answer to this question.  Indeed, rectified linear units haveonly begun to be widely used in the past few years.  The reason forthat recent adoption is empirical: a few people tried rectified linearunits, often on the basis of hunches or heuristic arguments*<span class="marginnote">*A  common justification is that $\max(0, z)$ doesn't saturate in the  limit of large $z$, unlike sigmoid neurons, and this helps rectified  linear units continue learning.  The argument is fine, as far it  goes, but it's hardly a detailed justification, more of a just-so  story.  Note that we discussed the problems with saturation back in  <a href="chap2.html#saturation">Chapter 2</a>.</span>.  They got goodresults classifying benchmark data sets, and the practice has spread.In an ideal world we'd have a theory telling us which activationfunction to pick for which application.  But at present we're a longway from such a world.  I should not be at all surprised if furthermajor improvements can be obtained by an even better choice ofactivation function.  And I also expect that in coming decades apowerful theory of activation functions will be developed.  Today, westill have to rely on poorly understood rules of thumb and experience.</p><p><strong>Expanding the training data:</strong> Another way we may hope toimprove our results is by algorithmically expanding the training data.A simple way of expanding the training data is to displace eachtraining image by a single pixel, either up one pixel, down one pixel,left one pixel, or right one pixel.  We can do this by running theprogram <tt>expand_mnist.py</tt> from the shell prompt*<span class="marginnote">*The code  for <tt>expand_mnist.py</tt> is available  <a href="https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/expand_mnist.py">here</a>.</span>:</p><p><div class="highlight"><pre><span></span> 
$ python expand_mnist.py
</pre></div>
 </p><p>Running this program takes the $50,000$ MNIST training images, andprepares an expanded training set, with $250,000$ training images.  Wecan then use those training images to train our network.  We'll usethe same network as above, with rectified linear units.  In my initialexperiments I reduced the number of training epochs - this madesense, since we're training with $5$ times as much data.  But, infact, expanding the data turned out to considerably reduce the effectof overfitting.  And so, after some experimentation, I eventually wentback to training for $60$ epochs.  In any case, let's train:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">expanded_training_data</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">network3</span><span class="o">.</span><span class="n">load_data_shared</span><span class="p">(</span>
        <span class="s2">&quot;../data/mnist_expanded.pkl.gz&quot;</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">12</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">40</span><span class="o">*</span><span class="mi">4</span><span class="o">*</span><span class="mi">4</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">expanded_training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.03</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="n">lmbda</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span>
</pre></div>
</p><p></p><p>Using the expanded training data I obtained a $99.37$ percent trainingaccuracy.  So this almost trivial change gives a substantialimprovement in classification accuracy.  Indeed, as we<a href="chap3.html#other_techniques_for_regularization">discussed  earlier</a> this idea of algorithmically expanding the data can betaken further.  Just to remind you of the flavour of some of theresults in that earlier discussion: in 2003 Simard, Steinkraus andPlatt*<span class="marginnote">*<a href="http://dx.doi.org/10.1109/ICDAR.2003.1227801">Best    Practices for Convolutional Neural Networks Applied to Visual    Document Analysis</a>, by Patrice Simard, Dave Steinkraus, and John  Platt (2003).</span> improved their MNIST performance to $99.6$ percentusing a neural network otherwise very similar to ours, using twoconvolutional-pooling layers, followed by a hidden fully-connectedlayer with $100$ neurons.  There were a few differences of detail intheir architecture - they didn't have the advantage of usingrectified linear units, for instance - but the key to their improvedperformance was expanding the training data.  They did this byrotating, translating, and skewing the MNIST training images.  Theyalso developed a process of "elastic distortion", a way of emulatingthe random oscillations hand muscles undergo when a person is writing.By combining all these processes they substantially increased theeffective size of their training data, and that's how they achieved$99.6$ percent accuracy.</p><p><h4><a name="problem_437600"></a><a href="chap6.html#problem_437600">Problem</a></h4><ul><li> The idea of convolutional layers is to behave in an invariant  way across images.  It may seem surprising, then, that our network  can learn more when all we've done is translate the input data.  Can  you explain why this is actually quite reasonable?</ul></p><p><strong>Inserting an extra fully-connected layer:</strong> Can we do evenbetter?  One possibility is to use exactly the same procedure asabove, but to expand the size of the fully-connected layer.  I triedwith $300$ and $1,000$ neurons, obtaining results of $99.46$ and$99.43$ percent, respectively.  That's interesting, but not really aconvincing win over the earlier result ($99.37$ percent).</p><p>What about adding an extra fully-connected layer?  Let's try insertingan extra fully-connected layer, so that we have two $100$-hiddenneuron fully-connected layers:<div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">12</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">40</span><span class="o">*</span><span class="mi">4</span><span class="o">*</span><span class="mi">4</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">)],</span> <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">expanded_training_data</span><span class="p">,</span> <span class="mi">60</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.03</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="n">lmbda</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span>
</pre></div>
</p><p>Doing this, I obtained a test accuracy of $99.43$ percent.  Again, theexpanded net isn't helping so much.  Running similar experiments withfully-connected layers containing $300$ and $1,000$ neurons yieldsresults of $99.48$ and $99.47$ percent.  That's encouraging, but stillfalls short of a really decisive win.</p><p><a name="final_conv"></a></p><p>What's going on here?  Is it that the expanded or extrafully-connected layers really don't help with MNIST?  Or might it bethat our network has the capacity to do better, but we're going aboutlearning the wrong way?  For instance, maybe we could use strongerregularization techniques to reduce the tendency to overfit.  Onepossibility is the<a href="chap3.html#other_techniques_for_regularization">dropout</a>technique introduced back in Chapter 3.  Recall that the basic idea ofdropout is to remove individual activations at random while trainingthe network.  This makes the model more robust to the loss ofindividual pieces of evidence, and thus less likely to rely onparticular idiosyncracies of the training data.  Let's try applyingdropout to the final fully-connected layers:</p><p><div class="highlight"><pre><span></span><span class="o">&gt;&gt;&gt;</span> <span class="n">net</span> <span class="o">=</span> <span class="n">Network</span><span class="p">([</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">ConvPoolLayer</span><span class="p">(</span><span class="n">image_shape</span><span class="o">=</span><span class="p">(</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">12</span><span class="p">),</span> 
                      <span class="n">filter_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">40</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">),</span> 
                      <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> 
                      <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span>
            <span class="n">n_in</span><span class="o">=</span><span class="mi">40</span><span class="o">*</span><span class="mi">4</span><span class="o">*</span><span class="mi">4</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.5</span><span class="p">),</span>
        <span class="n">FullyConnectedLayer</span><span class="p">(</span>
            <span class="n">n_in</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">ReLU</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.5</span><span class="p">),</span>
        <span class="n">SoftmaxLayer</span><span class="p">(</span><span class="n">n_in</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">n_out</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)],</span> 
        <span class="n">mini_batch_size</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">net</span><span class="o">.</span><span class="n">SGD</span><span class="p">(</span><span class="n">expanded_training_data</span><span class="p">,</span> <span class="mi">40</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="mf">0.03</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">)</span>
</pre></div>
</p><p>Using this, we obtain an accuracy of $99.60$ percent, which is asubstantial improvement over our earlier results, especially our mainbenchmark, the network with $100$ hidden neurons, where we achieved$99.37$ percent.</p><p>There are two changes worth noting.  </p><p>First, I reduced the number of training epochs to $40$: dropoutreduced overfitting, and so we learned faster.</p><p>Second, the fully-connected hidden layers have $1,000$ neurons, notthe $100$ used earlier. Of course, dropout effectively omits many ofthe neurons while training, so some expansion is to be expected.  Infact, I tried experiments with both $300$ and $1,000$ hidden neurons,and obtained (very slightly) better validation performance with$1,000$ hidden neurons.</p><p><strong>Using an ensemble of networks:</strong> An easy way to improveperformance still further is to create several neural networks, andthen get them to vote to determine the best classification.  Suppose,for example, that we trained $5$ different neural networks using theprescription above, with each achieving accuracies near to $99.6$percent.  Even though the networks would all have similar accuracies,they might well make different errors, due to the different randominitializations.  It's plausible that taking a vote amongst our $5$networks might yield a classification better than any individualnetwork.</p><p>This sounds too good to be true, but this kind of ensembling is acommon trick with both neural networks and other machine learningtechniques.  And it does in fact yield further improvements: we end upwith $99.67$ percent accuracy.  In other words, our ensemble ofnetworks classifies all but $33$ of the $10,000$ test imagescorrectly.  </p><p>The remaining errors in the test set are shown below.  The label inthe top right is the correct classification, according to the MNISTdata, while in the bottom right is the label output by our ensemble ofnets:</p><p><center><img src="images/ensemble_errors.png" width="580px"></center></p><p>It's worth looking through these in detail. The first two digits, a 6and a 5, are genuine errors by our ensemble.  However, they're alsounderstandable errors, the kind a human could plausibly make. That 6really does look a lot like a 0, and the 5 looks a lot like a 3.  Thethird image, supposedly an 8, actually looks to me more like a 9.  SoI'm siding with the network ensemble here: I think it's done a betterjob than whoever originally drew the digit.  On the other hand, thefourth image, the 6, really does seem to be classified badly by ournetworks.</p><p>And so on.  In most cases our networks' choices seem at leastplausible, and in some cases they've done a better job classifyingthan the original person did writing the digit.  Overall, our networksoffer exceptional performance, especially when you consider that theycorrectly classified 9,967 images which aren't shown.  In thatcontext, the few clear errors here seem quite understandable.  Even acareful human makes the occasional mistake.  And so I expect that onlyan extremely careful and methodical human would do much better.  Ournetwork is getting near to human performance.</p><p><strong>Why we only applied dropout to the fully-connected layers:</strong> Ifyou look carefully at the code above, you'll notice that we applieddropout only to the fully-connected section of the network, not to theconvolutional layers.  In principle we could apply a similar procedureto the convolutional layers.  But, in fact, there's no need: theconvolutional layers have considerable inbuilt resistance tooverfitting.  The reason is that the shared weights mean thatconvolutional filters are forced to learn from across the entireimage.  This makes them less likely to pick up on local idiosyncraciesin the training data.  And so there is less need to apply otherregularizers, such as dropout.</p><p><strong>Going further:</strong> It's possible to improve performance on MNISTstill further. Rodrigo Benenson has compiled an<a href="http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html">informative  summary page</a>, showing progress over the years, with links topapers.  Many of these papers use deep convolutional networks alonglines similar to the networks we've been using.  If you dig throughthe papers you'll find many interesting techniques, and you may enjoyimplementing some of them.  If you do so it's wise to startimplementation with a simple network that can be trained quickly,which will help you more rapidly understand what is going on.</p><p>For the most part, I won't try to survey this recent work.  But Ican't resist making one exception.  It's a 2010 paper by Cireșan,Meier, Gambardella, andSchmidhuber*<span class="marginnote">*<a href="http://arxiv.org/abs/1003.0358">Deep, Big,    Simple Neural Nets Excel on Handwritten Digit Recognition</a>, by Dan  Claudiu Cireșan, Ueli Meier, Luca Maria Gambardella, and Jürgen  Schmidhuber (2010).</span>.  What I like about this paper is how simple itis.  The network is a many-layer neural network, using onlyfully-connected layers (no convolutions).  Their most successfulnetwork had hidden layers containing $2,500$, $2,000$, $1,500$,$1,000$, and $500$ neurons, respectively.  They used ideas similar toSimard <em>et al</em> to expand their training data.  But apart fromthat, they used few other tricks, including no convolutional layers:it was a plain, vanilla network, of the kind that, with enoughpatience, could have been trained in the 1980s (if the MNIST data sethad existed), given enough computing power.  They achieved aclassification accuracy of $99.65$ percent, more or less the same asours.  The key was to use a very large, very deep network, and to usea GPU to speed up training.  This let them train for many epochs.They also took advantage of their long training times to graduallydecrease the learning rate from $10^{-3}$ to $10^{-6}$.  It's a funexercise to try to match these results using an architecture liketheirs.</p><p><strong>Why are we able to train?</strong>  We saw in <a href="chap5.html">the  last chapter</a> that there are fundamental obstructions to training indeep, many-layer neural networks.  In particular, we saw that thegradient tends to be quite unstable: as we move from the output layerto earlier layers the gradient tends to either vanish (the vanishinggradient problem) or explode (the exploding gradient problem).  Sincethe gradient is the signal we use to train, this causes problems.</p><p>How have we avoided those results?  </p><p>Of course, the answer is that we haven't avoided these results.Instead, we've done a few things that help us proceed anyway.  Inparticular: (1) Using convolutional layers greatly reduces the numberof parameters in those layers, making the learning problem mucheasier; (2) Using more powerful regularization techniques (notablydropout and convolutional layers) to reduce overfitting, which isotherwise more of a problem in more complex networks; (3) Usingrectified linear units instead of sigmoid neurons, to speed uptraining - empirically, often by a factor of $3$-$5$; (4) Using GPUsand being willing to train for a long period of time.  In particular,in our final experiments we trained for $40$ epochs using a data set$5$ times larger than the raw MNIST training data.  Earlier in thebook we mostly trained for $30$ epochs using just the raw trainingdata.  Combining factors (3) and (4) it's as though we've trained afactor perhaps $30$ times longer than before.</p><p>Your response may be "Is that it? Is that all we had to do to traindeep networks?  What's all the fuss about?"</p><p>Of course, we've used other ideas, too: making use of sufficientlylarge data sets (to help avoid overfitting); using the right costfunction (to<a href="chap3.html#the_cross\-entropy_cost_function">avoid a  learning slowdown</a>); using<a href="chap3.html#weight_initialization">good weight initializations</a>(also to avoid a learning slowdown, due to neuron saturation);<a href="chap3.html#other_techniques_for_regularization">algorithmically  expanding the training data</a>.  We discussed these and other ideas inearlier chapters, and have for the most part been able to reuse theseideas with little comment in this chapter.</p><p>With that said, this really is a rather simple set of ideas.  Simple,but powerful, when used in concert.  Getting started with deeplearning has turned out to be pretty easy!</p><p><strong>How deep are these networks, anyway?</strong> Counting theconvolutional-pooling layers as single layers, our final architecturehas $4$ hidden layers.  Does such a network really deserve to becalled a <em>deep</em> network?  Of course, $4$ hidden layers is manymore than in the shallow networks we studied earlier.  Most of thosenetworks only had a single hidden layer, or occasionally $2$ hiddenlayers.  On the other hand, as of 2015 state-of-the-art deep networkssometimes have dozens of hidden layers.  I've occasionally heardpeople adopt a deeper-than-thou attitude, holding that if you're notkeeping-up-with-the-Joneses in terms of number of hidden layers, thenyou're not really doing deep learning.  I'm not sympathetic to thisattitude, in part because it makes the definition of deep learninginto something which depends upon the result-of-the-moment. The realbreakthrough in deep learning was to realize that it's practical to gobeyond the shallow $1$- and $2$-hidden layer networks that dominatedwork until the mid-2000s.  That really was a significant breakthrough,opening up the exploration of much more expressive models.  But beyondthat, the number of layers is not of primary fundamental interest.Rather, the use of deeper networks is a tool to use to help achieveother goals - like better classification accuracies.</p><p><strong>A word on procedure:</strong> In this section, we've smoothly movedfrom single hidden-layer shallow networks to many-layer convolutionalnetworks.  It all seemed so easy!  We make a change and, for themost part, we get an improvement.  If you start experimenting, I canguarantee things won't always be so smooth.  The reason is that I'vepresented a cleaned-up narrative, omitting many experiments -including many failed experiments.  This cleaned-up narrative willhopefully help you get clear on the basic ideas.  But it also runs therisk of conveying an incomplete impression.  Getting a good, workingnetwork can involve a lot of trial and error, and occasionalfrustration.  In practice, you should expect to engage in quite a bitof experimentation.  To speed that process up you may find it helpfulto revisit Chapter 3's discussion of<a href="chap3.html#how_to_choose_a_neural_network's_hyper-parameters">how  to choose a neural network's hyper-parameters</a>, and perhaps also tolook at some of the further reading suggested in that section.</p><p></p><p></p><p></p><p><h3><a name="the_code_for_our_convolutional_networks"></a><a href="chap6.html#the_code_for_our_convolutional_networks">The code for our convolutional networks</a></h3></p><p>Alright, let's take a look at the code for our program,<tt>network3.py</tt>.  Structurally, it's similar to <tt>network2.py</tt>,the program we developed in <a href="chap3.html">Chapter 3</a>, although thedetails differ, due to the use of Theano.  We'll start by looking atthe <tt>FullyConnectedLayer</tt> class, which is similar to the layersstudied earlier in the book.  Here's the code (discussionbelow)*<span class="marginnote">*Note added November 2016: several readers have noted  that in the line initializing <tt>self.w</tt>, I set  <tt>scale=np.sqrt(1.0/n_out)</tt>, when the arguments of Chapter 3  suggest a better initialization may be  <tt>scale=np.sqrt(1.0/n_in)</tt>.  This was simply a mistake on my  part.  In an ideal world I'd rerun all the examples in this chapter  with the correct code. Still, I've moved on to other projects, so am  going to let the error go.</span>:</p><p><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FullyConnectedLayer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">sigmoid</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.0</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span> <span class="o">=</span> <span class="n">n_in</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_out</span> <span class="o">=</span> <span class="n">n_out</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span> <span class="o">=</span> <span class="n">activation_fn</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span> <span class="o">=</span> <span class="n">p_dropout</span>
        <span class="c1"># Initialize weights and biases</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">w</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span>
                <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span>
                    <span class="n">loc</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n_out</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">)),</span>
                <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;w&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">b</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">n_out</span><span class="p">,)),</span>
                       <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">]</span>

    <span class="k">def</span> <span class="nf">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inpt</span><span class="p">,</span> <span class="n">inpt_dropout</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt</span> <span class="o">=</span> <span class="n">inpt</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="p">(</span>
            <span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span><span class="o">*</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span> <span class="o">=</span> <span class="n">dropout_layer</span><span class="p">(</span>
            <span class="n">inpt_dropout</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">)),</span> <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="p">(</span>
            <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
        <span class="s2">&quot;Return the accuracy for the mini-batch.&quot;</span>
        <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span><span class="p">))</span>
</pre></div>
</p><p>Much of the <tt>__init__</tt> method is self-explanatory, but a fewremarks may help clarify the code.  As per usual, we randomlyinitialize the weights and biases as normal random variables withsuitable standard deviations.  The lines doing this look a littleforbidding.  However, most of the complication is just loading theweights and biases into what Theano calls shared variables.  Thisensures that these variables can be processed on the GPU, if one isavailable.  We won't get too much into the details of this.  If you'reinterested, you can dig into the<a href="http://deeplearning.net/software/theano/index.html">Theano  documentation</a>.  Note also that this weight and bias initializationis designed for the sigmoid activation function (as<a href="chap3.html#weight_initialization">discussed earlier</a>).Ideally, we'd initialize the weights and biases somewhat differentlyfor activation functions such as the tanh and rectified linearfunction.  This is discussed further in problems below.  The<tt>__init__</tt> method finishes with<tt>self.params = [self.w, self.b]</tt>.  This is a handy way to bundleup all the learnable parameters associated to the layer.  Later on,the <tt>Network.SGD</tt> method will use <tt>params</tt> attributes tofigure out what variables in a <tt>Network</tt> instance can learn.</p><p>The <tt>set_inpt</tt> method is used to set the input to the layer, andto compute the corresponding output.  I use the name <tt>inpt</tt>rather than <tt>input</tt> because <tt>input</tt> is a built-in functionin Python, and messing with built-ins tends to cause unpredictablebehavior and difficult-to-diagnose bugs.  Note that we actually setthe input in two separate ways: as <tt>self.inpt</tt> and<tt>self.inpt_dropout</tt>.  This is done because during training we maywant to use dropout.  If that's the case then we want to remove afraction <tt>self.p_dropout</tt> of the neurons.  That's what thefunction <tt>dropout_layer</tt> in the second-last line of the<tt>set_inpt</tt> method is doing.  So <tt>self.inpt_dropout</tt> and<tt>self.output_dropout</tt> are used during training, while<tt>self.inpt</tt> and <tt>self.output</tt> are used for all otherpurposes, e.g., evaluating accuracy on the validation and test data.</p><p>The <tt>ConvPoolLayer</tt> and <tt>SoftmaxLayer</tt> class definitions aresimilar to <tt>FullyConnectedLayer</tt>.  Indeed, they're so close thatI won't excerpt the code here.  If you're interested you can look atthe full listing for <tt>network3.py</tt>, later in this section.</p><p>However, a couple of minor differences of detail are worth mentioning.Most obviously, in both <tt>ConvPoolLayer</tt> and <tt>SoftmaxLayer</tt>we compute the output activations in the way appropriate to that layertype.  Fortunately, Theano makes that easy, providing built-inoperations to compute convolutions, max-pooling, and the softmaxfunction.</p><p>Less obviously, when we <a href="chap3.html#softmax">introduced the  softmax layer</a>, we never discussed how to initialize the weights andbiases.  Elsewhere we've argued that for sigmoid layers we shouldinitialize the weights using suitably parameterized normal randomvariables.  But that heuristic argument was specific to sigmoidneurons (and, with some amendment, to tanh neurons).  However, there'sno particular reason the argument should apply to softmax layers.  Sothere's no <em>a priori</em> reason to apply that initialization again.Rather than do that, I shall initialize all the weights and biases tobe $0$.  This is a rather <em>ad hoc</em> procedure, but works wellenough in practice.</p><p>Okay, we've looked at all the layer classes.  What about the<tt>Network</tt> class?  Let's start by looking at the <tt>__init__</tt>method:</p><p><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Network</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Takes a list of `layers`, describing the network architecture, and</span>
<span class="sd">        a value for the `mini_batch_size` to be used during training</span>
<span class="sd">        by stochastic gradient descent.</span>

<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="o">=</span> <span class="n">layers</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span> <span class="o">=</span> <span class="n">mini_batch_size</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="n">param</span> <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">layer</span><span class="o">.</span><span class="n">params</span><span class="p">]</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">matrix</span><span class="p">(</span><span class="s2">&quot;x&quot;</span><span class="p">)</span>  
        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">ivector</span><span class="p">(</span><span class="s2">&quot;y&quot;</span><span class="p">)</span>
        <span class="n">init_layer</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
        <span class="n">init_layer</span><span class="o">.</span><span class="n">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">)</span>
        <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">)):</span>
            <span class="n">prev_layer</span><span class="p">,</span> <span class="n">layer</span>  <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="n">j</span><span class="p">]</span>
            <span class="n">layer</span><span class="o">.</span><span class="n">set_inpt</span><span class="p">(</span>
                <span class="n">prev_layer</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">prev_layer</span><span class="o">.</span><span class="n">output_dropout</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">output</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">output_dropout</span>
</pre></div>
</p><p>Most of this is self-explanatory, or nearly so.  The line<tt>self.params = [param for layer in ...]</tt> bundles up theparameters for each layer into a single list.  As anticipated above,the <tt>Network.SGD</tt> method will use <tt>self.params</tt> to figureout what variables in the <tt>Network</tt> can learn.  The lines<tt>self.x = T.matrix("x")</tt> and <tt>self.y = T.ivector("y")</tt>define Theano symbolic variables named <tt>x</tt> and <tt>y</tt>.  Thesewill be used to represent the input and desired output from thenetwork.</p><p>Now, this isn't a Theano tutorial, and so we won't get too deeply intowhat it means that these are symbolic variables*<span class="marginnote">*The  <a href="http://deeplearning.net/software/theano/index.html">Theano    documentation</a> provides a good introduction to Theano.  And if you  get stuck, you may find it helpful to look at one of the other  tutorials available online.  For instance,  <a href="http://nbviewer.ipython.org/github/craffel/theano-tutorial/blob/master/Theano%20Tutorial.ipynb">this    tutorial</a> covers many basics.</span>.  But the rough idea is that theserepresent mathematical variables, <em>not</em> explicit values.  We cando all the usual things one would do with such variables: add,subtract, and multiply them, apply functions, and so on.  Indeed,Theano provides many ways of manipulating such symbolic variables,doing things like convolutions, max-pooling, and so on.  But the bigwin is the ability to do fast symbolic differentiation, using a verygeneral form of the backpropagation algorithm.  This is extremelyuseful for applying stochastic gradient descent to a wide variety ofnetwork architectures.  In particular, the next few lines of codedefine symbolic outputs from the network.  We start by setting theinput to the initial layer, with the line</p><p><div class="highlight"><pre><span></span>        <span class="n">init_layer</span><span class="o">.</span><span class="n">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">)</span>
</pre></div>
</p><p>Note that the inputs are set one mini-batch at a time, which is whythe mini-batch size is there.  Note also that we pass the input<tt>self.x</tt> in twice: this is because we may use the network in twodifferent ways (with or without dropout).  The <tt>for</tt> loop thenpropagates the symbolic variable <tt>self.x</tt> forward through thelayers of the <tt>Network</tt>. This allows us to define the final<tt>output</tt> and <tt>output_dropout</tt> attributes, which symbolicallyrepresent the output from the <tt>Network</tt>.</p><p>Now that we've understood how a <tt>Network</tt> is initialized, let'slook at how it is trained, using the <tt>SGD</tt> method.  The codelooks lengthy, but its structure is actually rather simple.Explanatory comments after the code.</p><p><div class="highlight"><pre><span></span>    <span class="k">def</span> <span class="nf">SGD</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">training_data</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="n">eta</span><span class="p">,</span> 
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="n">lmbda</span><span class="o">=</span><span class="mf">0.0</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Train the network using mini-batch stochastic gradient descent.&quot;&quot;&quot;</span>
        <span class="n">training_x</span><span class="p">,</span> <span class="n">training_y</span> <span class="o">=</span> <span class="n">training_data</span>
        <span class="n">validation_x</span><span class="p">,</span> <span class="n">validation_y</span> <span class="o">=</span> <span class="n">validation_data</span>
        <span class="n">test_x</span><span class="p">,</span> <span class="n">test_y</span> <span class="o">=</span> <span class="n">test_data</span>

        <span class="c1"># compute number of minibatches for training, validation and testing</span>
        <span class="n">num_training_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">training_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>
        <span class="n">num_validation_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">validation_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>
        <span class="n">num_test_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">test_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>

        <span class="c1"># define the (regularized) cost function, symbolic gradients, and updates</span>
        <span class="n">l2_norm_squared</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">([(</span><span class="n">layer</span><span class="o">.</span><span class="n">w</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">])</span>
        <span class="n">cost</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">cost</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">+</span>\
               <span class="mf">0.5</span><span class="o">*</span><span class="n">lmbda</span><span class="o">*</span><span class="n">l2_norm_squared</span><span class="o">/</span><span class="n">num_training_batches</span>
        <span class="n">grads</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">cost</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">)</span>
        <span class="n">updates</span> <span class="o">=</span> <span class="p">[(</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span><span class="o">-</span><span class="n">eta</span><span class="o">*</span><span class="n">grad</span><span class="p">)</span> 
                   <span class="k">for</span> <span class="n">param</span><span class="p">,</span> <span class="n">grad</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span> <span class="n">grads</span><span class="p">)]</span>

        <span class="c1"># define functions to train a mini-batch, and to compute the</span>
        <span class="c1"># accuracy in validation and test mini-batches.</span>
        <span class="n">i</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">lscalar</span><span class="p">()</span> <span class="c1"># mini-batch index</span>
        <span class="n">train_mb</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">cost</span><span class="p">,</span> <span class="n">updates</span><span class="o">=</span><span class="n">updates</span><span class="p">,</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span>
                <span class="n">training_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span> 
                <span class="n">training_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="n">validate_mb_accuracy</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">),</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span> 
                <span class="n">validation_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span> 
                <span class="n">validation_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="n">test_mb_accuracy</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">),</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span> 
                <span class="n">test_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span> 
                <span class="n">test_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">test_mb_predictions</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">y_out</span><span class="p">,</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span> 
                <span class="n">test_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="c1"># Do the actual training</span>
        <span class="n">best_validation_accuracy</span> <span class="o">=</span> <span class="mf">0.0</span>
        <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">epochs</span><span class="p">):</span>
            <span class="k">for</span> <span class="n">minibatch_index</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_training_batches</span><span class="p">):</span>
                <span class="n">iteration</span> <span class="o">=</span> <span class="n">num_training_batches</span><span class="o">*</span><span class="n">epoch</span><span class="o">+</span><span class="n">minibatch_index</span>
                <span class="k">if</span> <span class="n">iteration</span> 
                    <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Training mini-batch number {0}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">iteration</span><span class="p">))</span>
                <span class="n">cost_ij</span> <span class="o">=</span> <span class="n">train_mb</span><span class="p">(</span><span class="n">minibatch_index</span><span class="p">)</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">iteration</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> 
                    <span class="n">validation_accuracy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span>
                        <span class="p">[</span><span class="n">validate_mb_accuracy</span><span class="p">(</span><span class="n">j</span><span class="p">)</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_validation_batches</span><span class="p">)])</span>
                    <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Epoch {0}: validation accuracy {1:.2</span>
                        <span class="n">epoch</span><span class="p">,</span> <span class="n">validation_accuracy</span><span class="p">))</span>
                    <span class="k">if</span> <span class="n">validation_accuracy</span> <span class="o">&gt;=</span> <span class="n">best_validation_accuracy</span><span class="p">:</span>
                        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;This is the best validation accuracy to date.&quot;</span><span class="p">)</span>
                        <span class="n">best_validation_accuracy</span> <span class="o">=</span> <span class="n">validation_accuracy</span>
                        <span class="n">best_iteration</span> <span class="o">=</span> <span class="n">iteration</span>
                        <span class="k">if</span> <span class="n">test_data</span><span class="p">:</span>
                            <span class="n">test_accuracy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span>
                                <span class="p">[</span><span class="n">test_mb_accuracy</span><span class="p">(</span><span class="n">j</span><span class="p">)</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_test_batches</span><span class="p">)])</span>
                            <span class="k">print</span><span class="p">(</span><span class="s1">&#39;The corresponding test accuracy is {0:.2</span>
                                <span class="n">test_accuracy</span><span class="p">))</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Finished training network.&quot;</span><span class="p">)</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Best validation accuracy of {0:.2</span>
            <span class="n">best_validation_accuracy</span><span class="p">,</span> <span class="n">best_iteration</span><span class="p">))</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Corresponding test accuracy of {0:.2</span>
</pre></div>
</p><p>The first few lines are straightforward, separating the datasets into$x$ and $y$ components, and computing the number of mini-batches usedin each dataset.  The next few lines are more interesting, and showsome of what makes Theano fun to work with.  Let's explicitly excerptthe lines here:</p><p><div class="highlight"><pre><span></span>        <span class="c1"># define the (regularized) cost function, symbolic gradients, and updates</span>
        <span class="n">l2_norm_squared</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">([(</span><span class="n">layer</span><span class="o">.</span><span class="n">w</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">])</span>
        <span class="n">cost</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">cost</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">+</span>\
               <span class="mf">0.5</span><span class="o">*</span><span class="n">lmbda</span><span class="o">*</span><span class="n">l2_norm_squared</span><span class="o">/</span><span class="n">num_training_batches</span>
        <span class="n">grads</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">cost</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">)</span>
        <span class="n">updates</span> <span class="o">=</span> <span class="p">[(</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span><span class="o">-</span><span class="n">eta</span><span class="o">*</span><span class="n">grad</span><span class="p">)</span> 
                   <span class="k">for</span> <span class="n">param</span><span class="p">,</span> <span class="n">grad</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span> <span class="n">grads</span><span class="p">)]</span>
</pre></div>
</p><p>In these lines we symbolically set up the regularized log-likelihoodcost function, compute the corresponding derivatives in the gradientfunction, as well as the corresponding parameter updates.  Theano letsus achieve all of this in just these few lines.  The only thing hiddenis that computing the <tt>cost</tt> involves a call to the <tt>cost</tt>method for the output layer; that code is elsewhere in<tt>network3.py</tt>.  But that code is short and simple, anyway.  Withall these things defined, the stage is set to define the<tt>train_mb</tt> function, a Theano symbolic function which uses the<tt>updates</tt> to update the <tt>Network</tt> parameters, given amini-batch index.  Similarly, <tt>validate_mb_accuracy</tt> and<tt>test_mb_accuracy</tt> compute the accuracy of the <tt>Network</tt> onany given mini-batch of validation or test data.  By averaging overthese functions, we will be able to compute accuracies on the entirevalidation and test data sets.</p><p>The remainder of the <tt>SGD</tt> method is self-explanatory - wesimply iterate over the epochs, repeatedly training the network onmini-batches of training data, and computing the validation and testaccuracies.  </p><p>Okay, we've now understood the most important pieces of code in<tt>network3.py</tt>.  Let's take a brief look at the entire program.You don't need to read through this in detail, but you may enjoyglancing over it, and perhaps diving down into any pieces that strikeyour fancy.  The best way to really understand it is, of course, bymodifying it, adding extra features, or refactoring anything you thinkcould be done more elegantly.  After the code, there are some problemswhich contain a few starter suggestions for things to do.  Here's thecode*<span class="marginnote">*Using Theano on a GPU can be a little tricky.  In  particular, it's easy to make the mistake of pulling data off the  GPU, which can slow things down a lot.  I've tried to avoid this.  With that said, this code can certainly be sped up quite a bit  further with careful optimization of Theano's configuration.  See  the Theano documentation for more details.</span>:</p><p><div class="highlight"><pre><span></span><span class="sd">&quot;&quot;&quot;network3.py</span>
<span class="sd">~~~~~~~~~~~~~~</span>

<span class="sd">A Theano-based program for training and running simple neural</span>
<span class="sd">networks.</span>

<span class="sd">Supports several layer types (fully connected, convolutional, max</span>
<span class="sd">pooling, softmax), and activation functions (sigmoid, tanh, and</span>
<span class="sd">rectified linear units, with more easily added).</span>

<span class="sd">When run on a CPU, this program is much faster than network.py and</span>
<span class="sd">network2.py.  However, unlike network.py and network2.py it can also</span>
<span class="sd">be run on a GPU, which makes it faster still.</span>

<span class="sd">Because the code is based on Theano, the code is different in many</span>
<span class="sd">ways from network.py and network2.py.  However, where possible I have</span>
<span class="sd">tried to maintain consistency with the earlier programs.  In</span>
<span class="sd">particular, the API is similar to network2.py.  Note that I have</span>
<span class="sd">focused on making the code simple, easily readable, and easily</span>
<span class="sd">modifiable.  It is not optimized, and omits many desirable features.</span>

<span class="sd">This program incorporates ideas from the Theano documentation on</span>
<span class="sd">convolutional neural nets (notably,</span>
<span class="sd">http://deeplearning.net/tutorial/lenet.html ), from Misha Denil&#39;s</span>
<span class="sd">implementation of dropout (https://github.com/mdenil/dropout ), and</span>
<span class="sd">from Chris Olah (http://colah.github.io ).</span>

<span class="sd">Written for Theano 0.6 and 0.7, needs some changes for more recent</span>
<span class="sd">versions of Theano.</span>

<span class="sd">&quot;&quot;&quot;</span>

<span class="c1">#### Libraries</span>
<span class="c1"># Standard library</span>
<span class="kn">import</span> <span class="nn">cPickle</span>
<span class="kn">import</span> <span class="nn">gzip</span>

<span class="c1"># Third-party libraries</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">theano</span>
<span class="kn">import</span> <span class="nn">theano.tensor</span> <span class="kn">as</span> <span class="nn">T</span>
<span class="kn">from</span> <span class="nn">theano.tensor.nnet</span> <span class="kn">import</span> <span class="n">conv</span>
<span class="kn">from</span> <span class="nn">theano.tensor.nnet</span> <span class="kn">import</span> <span class="n">softmax</span>
<span class="kn">from</span> <span class="nn">theano.tensor</span> <span class="kn">import</span> <span class="n">shared_randomstreams</span>
<span class="kn">from</span> <span class="nn">theano.tensor.signal</span> <span class="kn">import</span> <span class="n">downsample</span>

<span class="c1"># Activation functions for neurons</span>
<span class="k">def</span> <span class="nf">linear</span><span class="p">(</span><span class="n">z</span><span class="p">):</span> <span class="k">return</span> <span class="n">z</span>
<span class="k">def</span> <span class="nf">ReLU</span><span class="p">(</span><span class="n">z</span><span class="p">):</span> <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">maximum</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">z</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">theano.tensor.nnet</span> <span class="kn">import</span> <span class="n">sigmoid</span>
<span class="kn">from</span> <span class="nn">theano.tensor</span> <span class="kn">import</span> <span class="n">tanh</span>


<span class="c1">#### Constants</span>
<span class="n">GPU</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">if</span> <span class="n">GPU</span><span class="p">:</span>
    <span class="k">print</span> <span class="s2">&quot;Trying to run under a GPU.  If this is not desired, then modify &quot;</span><span class="o">+</span>\
        <span class="s2">&quot;network3.py</span><span class="se">\n</span><span class="s2">to set the GPU flag to False.&quot;</span>
    <span class="k">try</span><span class="p">:</span> <span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">device</span> <span class="o">=</span> <span class="s1">&#39;gpu&#39;</span>
    <span class="k">except</span><span class="p">:</span> <span class="k">pass</span> <span class="c1"># it&#39;s already set</span>
    <span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span> <span class="o">=</span> <span class="s1">&#39;float32&#39;</span>
<span class="k">else</span><span class="p">:</span>
    <span class="k">print</span> <span class="s2">&quot;Running with a CPU.  If this is not desired, then the modify &quot;</span><span class="o">+</span>\
        <span class="s2">&quot;network3.py to set</span><span class="se">\n</span><span class="s2">the GPU flag to True.&quot;</span>

<span class="c1">#### Load the MNIST data</span>
<span class="k">def</span> <span class="nf">load_data_shared</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="s2">&quot;../data/mnist.pkl.gz&quot;</span><span class="p">):</span>
    <span class="n">f</span> <span class="o">=</span> <span class="n">gzip</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;rb&#39;</span><span class="p">)</span>
    <span class="n">training_data</span><span class="p">,</span> <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span> <span class="o">=</span> <span class="n">cPickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
    <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
    <span class="k">def</span> <span class="nf">shared</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Place the data into shared variables.  This allows Theano to copy</span>
<span class="sd">        the data to the GPU, if one is available.</span>

<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">shared_x</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="n">shared_y</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">shared_x</span><span class="p">,</span> <span class="n">T</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">shared_y</span><span class="p">,</span> <span class="s2">&quot;int32&quot;</span><span class="p">)</span>
    <span class="k">return</span> <span class="p">[</span><span class="n">shared</span><span class="p">(</span><span class="n">training_data</span><span class="p">),</span> <span class="n">shared</span><span class="p">(</span><span class="n">validation_data</span><span class="p">),</span> <span class="n">shared</span><span class="p">(</span><span class="n">test_data</span><span class="p">)]</span>

<span class="c1">#### Main class used to construct and train networks</span>
<span class="k">class</span> <span class="nc">Network</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Takes a list of `layers`, describing the network architecture, and</span>
<span class="sd">        a value for the `mini_batch_size` to be used during training</span>
<span class="sd">        by stochastic gradient descent.</span>

<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="o">=</span> <span class="n">layers</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span> <span class="o">=</span> <span class="n">mini_batch_size</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="n">param</span> <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="k">for</span> <span class="n">param</span> <span class="ow">in</span> <span class="n">layer</span><span class="o">.</span><span class="n">params</span><span class="p">]</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">matrix</span><span class="p">(</span><span class="s2">&quot;x&quot;</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">ivector</span><span class="p">(</span><span class="s2">&quot;y&quot;</span><span class="p">)</span>
        <span class="n">init_layer</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
        <span class="n">init_layer</span><span class="o">.</span><span class="n">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">)</span>
        <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">)):</span>
            <span class="n">prev_layer</span><span class="p">,</span> <span class="n">layer</span>  <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="n">j</span><span class="p">]</span>
            <span class="n">layer</span><span class="o">.</span><span class="n">set_inpt</span><span class="p">(</span>
                <span class="n">prev_layer</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">prev_layer</span><span class="o">.</span><span class="n">output_dropout</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">output</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">output_dropout</span>

    <span class="k">def</span> <span class="nf">SGD</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">training_data</span><span class="p">,</span> <span class="n">epochs</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">,</span> <span class="n">eta</span><span class="p">,</span>
            <span class="n">validation_data</span><span class="p">,</span> <span class="n">test_data</span><span class="p">,</span> <span class="n">lmbda</span><span class="o">=</span><span class="mf">0.0</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Train the network using mini-batch stochastic gradient descent.&quot;&quot;&quot;</span>
        <span class="n">training_x</span><span class="p">,</span> <span class="n">training_y</span> <span class="o">=</span> <span class="n">training_data</span>
        <span class="n">validation_x</span><span class="p">,</span> <span class="n">validation_y</span> <span class="o">=</span> <span class="n">validation_data</span>
        <span class="n">test_x</span><span class="p">,</span> <span class="n">test_y</span> <span class="o">=</span> <span class="n">test_data</span>

        <span class="c1"># compute number of minibatches for training, validation and testing</span>
        <span class="n">num_training_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">training_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>
        <span class="n">num_validation_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">validation_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>
        <span class="n">num_test_batches</span> <span class="o">=</span> <span class="n">size</span><span class="p">(</span><span class="n">test_data</span><span class="p">)</span><span class="o">/</span><span class="n">mini_batch_size</span>

        <span class="c1"># define the (regularized) cost function, symbolic gradients, and updates</span>
        <span class="n">l2_norm_squared</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">([(</span><span class="n">layer</span><span class="o">.</span><span class="n">w</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">])</span>
        <span class="n">cost</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">cost</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">+</span>\
               <span class="mf">0.5</span><span class="o">*</span><span class="n">lmbda</span><span class="o">*</span><span class="n">l2_norm_squared</span><span class="o">/</span><span class="n">num_training_batches</span>
        <span class="n">grads</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">grad</span><span class="p">(</span><span class="n">cost</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">)</span>
        <span class="n">updates</span> <span class="o">=</span> <span class="p">[(</span><span class="n">param</span><span class="p">,</span> <span class="n">param</span><span class="o">-</span><span class="n">eta</span><span class="o">*</span><span class="n">grad</span><span class="p">)</span>
                   <span class="k">for</span> <span class="n">param</span><span class="p">,</span> <span class="n">grad</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span> <span class="n">grads</span><span class="p">)]</span>

        <span class="c1"># define functions to train a mini-batch, and to compute the</span>
        <span class="c1"># accuracy in validation and test mini-batches.</span>
        <span class="n">i</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">lscalar</span><span class="p">()</span> <span class="c1"># mini-batch index</span>
        <span class="n">train_mb</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">cost</span><span class="p">,</span> <span class="n">updates</span><span class="o">=</span><span class="n">updates</span><span class="p">,</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span>
                <span class="n">training_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span>
                <span class="n">training_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="n">validate_mb_accuracy</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">),</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span>
                <span class="n">validation_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span>
                <span class="n">validation_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="n">test_mb_accuracy</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">),</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span>
                <span class="n">test_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">],</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">:</span>
                <span class="n">test_y</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">test_mb_predictions</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">function</span><span class="p">(</span>
            <span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">y_out</span><span class="p">,</span>
            <span class="n">givens</span><span class="o">=</span><span class="p">{</span>
                <span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="p">:</span>
                <span class="n">test_x</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">:</span> <span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">mini_batch_size</span><span class="p">]</span>
            <span class="p">})</span>
        <span class="c1"># Do the actual training</span>
        <span class="n">best_validation_accuracy</span> <span class="o">=</span> <span class="mf">0.0</span>
        <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">epochs</span><span class="p">):</span>
            <span class="k">for</span> <span class="n">minibatch_index</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_training_batches</span><span class="p">):</span>
                <span class="n">iteration</span> <span class="o">=</span> <span class="n">num_training_batches</span><span class="o">*</span><span class="n">epoch</span><span class="o">+</span><span class="n">minibatch_index</span>
                <span class="k">if</span> <span class="n">iteration</span> <span class="o">%</span> <span class="mi">1000</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
                    <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Training mini-batch number {0}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">iteration</span><span class="p">))</span>
                <span class="n">cost_ij</span> <span class="o">=</span> <span class="n">train_mb</span><span class="p">(</span><span class="n">minibatch_index</span><span class="p">)</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">iteration</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="n">num_training_batches</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
                    <span class="n">validation_accuracy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span>
                        <span class="p">[</span><span class="n">validate_mb_accuracy</span><span class="p">(</span><span class="n">j</span><span class="p">)</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_validation_batches</span><span class="p">)])</span>
                    <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Epoch {0}: validation accuracy {1:.2%}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
                        <span class="n">epoch</span><span class="p">,</span> <span class="n">validation_accuracy</span><span class="p">))</span>
                    <span class="k">if</span> <span class="n">validation_accuracy</span> <span class="o">&gt;=</span> <span class="n">best_validation_accuracy</span><span class="p">:</span>
                        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;This is the best validation accuracy to date.&quot;</span><span class="p">)</span>
                        <span class="n">best_validation_accuracy</span> <span class="o">=</span> <span class="n">validation_accuracy</span>
                        <span class="n">best_iteration</span> <span class="o">=</span> <span class="n">iteration</span>
                        <span class="k">if</span> <span class="n">test_data</span><span class="p">:</span>
                            <span class="n">test_accuracy</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span>
                                <span class="p">[</span><span class="n">test_mb_accuracy</span><span class="p">(</span><span class="n">j</span><span class="p">)</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_test_batches</span><span class="p">)])</span>
                            <span class="k">print</span><span class="p">(</span><span class="s1">&#39;The corresponding test accuracy is {0:.2%}&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
                                <span class="n">test_accuracy</span><span class="p">))</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Finished training network.&quot;</span><span class="p">)</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Best validation accuracy of {0:.2%} obtained at iteration {1}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
            <span class="n">best_validation_accuracy</span><span class="p">,</span> <span class="n">best_iteration</span><span class="p">))</span>
        <span class="k">print</span><span class="p">(</span><span class="s2">&quot;Corresponding test accuracy of {0:.2%}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">test_accuracy</span><span class="p">))</span>

<span class="c1">#### Define layer types</span>

<span class="k">class</span> <span class="nc">ConvPoolLayer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;Used to create a combination of a convolutional and a max-pooling</span>
<span class="sd">    layer.  A more sophisticated implementation would separate the</span>
<span class="sd">    two, but for our purposes we&#39;ll always use them together, and it</span>
<span class="sd">    simplifies the code, so it makes sense to combine them.</span>

<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filter_shape</span><span class="p">,</span> <span class="n">image_shape</span><span class="p">,</span> <span class="n">poolsize</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span>
                 <span class="n">activation_fn</span><span class="o">=</span><span class="n">sigmoid</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;`filter_shape` is a tuple of length 4, whose entries are the number</span>
<span class="sd">        of filters, the number of input feature maps, the filter height, and the</span>
<span class="sd">        filter width.</span>

<span class="sd">        `image_shape` is a tuple of length 4, whose entries are the</span>
<span class="sd">        mini-batch size, the number of input feature maps, the image</span>
<span class="sd">        height, and the image width.</span>

<span class="sd">        `poolsize` is a tuple of length 2, whose entries are the y and</span>
<span class="sd">        x pooling sizes.</span>

<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">filter_shape</span> <span class="o">=</span> <span class="n">filter_shape</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">image_shape</span> <span class="o">=</span> <span class="n">image_shape</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">poolsize</span> <span class="o">=</span> <span class="n">poolsize</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="o">=</span><span class="n">activation_fn</span>
        <span class="c1"># initialize weights and biases</span>
        <span class="n">n_out</span> <span class="o">=</span> <span class="p">(</span><span class="n">filter_shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">prod</span><span class="p">(</span><span class="n">filter_shape</span><span class="p">[</span><span class="mi">2</span><span class="p">:])</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">prod</span><span class="p">(</span><span class="n">poolsize</span><span class="p">))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">w</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span>
                <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n_out</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="n">filter_shape</span><span class="p">),</span>
                <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">b</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span>
                <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">filter_shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],)),</span>
                <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">]</span>

    <span class="k">def</span> <span class="nf">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inpt</span><span class="p">,</span> <span class="n">inpt_dropout</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt</span> <span class="o">=</span> <span class="n">inpt</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">image_shape</span><span class="p">)</span>
        <span class="n">conv_out</span> <span class="o">=</span> <span class="n">conv</span><span class="o">.</span><span class="n">conv2d</span><span class="p">(</span>
            <span class="nb">input</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt</span><span class="p">,</span> <span class="n">filters</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">,</span> <span class="n">filter_shape</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filter_shape</span><span class="p">,</span>
            <span class="n">image_shape</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">image_shape</span><span class="p">)</span>
        <span class="n">pooled_out</span> <span class="o">=</span> <span class="n">downsample</span><span class="o">.</span><span class="n">max_pool_2d</span><span class="p">(</span>
            <span class="nb">input</span><span class="o">=</span><span class="n">conv_out</span><span class="p">,</span> <span class="n">ds</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">poolsize</span><span class="p">,</span> <span class="n">ignore_border</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="p">(</span>
            <span class="n">pooled_out</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="o">.</span><span class="n">dimshuffle</span><span class="p">(</span><span class="s1">&#39;x&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s1">&#39;x&#39;</span><span class="p">,</span> <span class="s1">&#39;x&#39;</span><span class="p">))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="c1"># no dropout in the convolutional layers</span>

<span class="k">class</span> <span class="nc">FullyConnectedLayer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">,</span> <span class="n">activation_fn</span><span class="o">=</span><span class="n">sigmoid</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.0</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span> <span class="o">=</span> <span class="n">n_in</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_out</span> <span class="o">=</span> <span class="n">n_out</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span> <span class="o">=</span> <span class="n">activation_fn</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span> <span class="o">=</span> <span class="n">p_dropout</span>
        <span class="c1"># Initialize weights and biases</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">w</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span>
                <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span>
                    <span class="n">loc</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n_out</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">)),</span>
                <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;w&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">b</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">scale</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="n">n_out</span><span class="p">,)),</span>
                       <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">]</span>

    <span class="k">def</span> <span class="nf">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inpt</span><span class="p">,</span> <span class="n">inpt_dropout</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt</span> <span class="o">=</span> <span class="n">inpt</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="p">(</span>
            <span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span><span class="o">*</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span> <span class="o">=</span> <span class="n">dropout_layer</span><span class="p">(</span>
            <span class="n">inpt_dropout</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">)),</span> <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">activation_fn</span><span class="p">(</span>
            <span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
        <span class="s2">&quot;Return the accuracy for the mini-batch.&quot;</span>
        <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span><span class="p">))</span>

<span class="k">class</span> <span class="nc">SoftmaxLayer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">,</span> <span class="n">p_dropout</span><span class="o">=</span><span class="mf">0.0</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span> <span class="o">=</span> <span class="n">n_in</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">n_out</span> <span class="o">=</span> <span class="n">n_out</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span> <span class="o">=</span> <span class="n">p_dropout</span>
        <span class="c1"># Initialize weights and biases</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">w</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n_in</span><span class="p">,</span> <span class="n">n_out</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;w&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">b</span> <span class="o">=</span> <span class="n">theano</span><span class="o">.</span><span class="n">shared</span><span class="p">(</span>
            <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n_out</span><span class="p">,),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">),</span>
            <span class="n">name</span><span class="o">=</span><span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">]</span>

    <span class="k">def</span> <span class="nf">set_inpt</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inpt</span><span class="p">,</span> <span class="n">inpt_dropout</span><span class="p">,</span> <span class="n">mini_batch_size</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt</span> <span class="o">=</span> <span class="n">inpt</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">))</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">((</span><span class="mi">1</span><span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span><span class="o">*</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span> <span class="o">=</span> <span class="n">T</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span> <span class="o">=</span> <span class="n">dropout_layer</span><span class="p">(</span>
            <span class="n">inpt_dropout</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">mini_batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_in</span><span class="p">)),</span> <span class="bp">self</span><span class="o">.</span><span class="n">p_dropout</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">inpt_dropout</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">w</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">cost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">net</span><span class="p">):</span>
        <span class="s2">&quot;Return the log-likelihood cost.&quot;</span>
        <span class="k">return</span> <span class="o">-</span><span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output_dropout</span><span class="p">)[</span><span class="n">T</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="n">net</span><span class="o">.</span><span class="n">y</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">net</span><span class="o">.</span><span class="n">y</span><span class="p">])</span>

    <span class="k">def</span> <span class="nf">accuracy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
        <span class="s2">&quot;Return the accuracy for the mini-batch.&quot;</span>
        <span class="k">return</span> <span class="n">T</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">T</span><span class="o">.</span><span class="n">eq</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_out</span><span class="p">))</span>


<span class="c1">#### Miscellanea</span>
<span class="k">def</span> <span class="nf">size</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
    <span class="s2">&quot;Return the size of the dataset `data`.&quot;</span>
    <span class="k">return</span> <span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">get_value</span><span class="p">(</span><span class="n">borrow</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>

<span class="k">def</span> <span class="nf">dropout_layer</span><span class="p">(</span><span class="n">layer</span><span class="p">,</span> <span class="n">p_dropout</span><span class="p">):</span>
    <span class="n">srng</span> <span class="o">=</span> <span class="n">shared_randomstreams</span><span class="o">.</span><span class="n">RandomStreams</span><span class="p">(</span>
        <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">RandomState</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">999999</span><span class="p">))</span>
    <span class="n">mask</span> <span class="o">=</span> <span class="n">srng</span><span class="o">.</span><span class="n">binomial</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">p</span><span class="o">=</span><span class="mi">1</span><span class="o">-</span><span class="n">p_dropout</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">layer</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">layer</span><span class="o">*</span><span class="n">T</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">mask</span><span class="p">,</span> <span class="n">theano</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">floatX</span><span class="p">)</span>
</pre></div>
</p><p><h4><a name="problems_269956"></a><a href="chap6.html#problems_269956">Problems</a></h4><ul></p><p><li> At present, the <tt>SGD</tt> method requires the user to manually  choose the number of epochs to train for.  Earlier in the book we  discussed an automated way of selecting the number of epochs to  train for, known as <a href="chap3.html#early_stopping">early    stopping</a>.  Modify <tt>network3.py</tt> to implement early stopping.</p><p><li> Add a <tt>Network</tt> method to return the accuracy on an  arbitrary data set.</p><p><li> Modify the <tt>SGD</tt> method to allow the learning rate $\eta$  to be a function of the epoch number.  <em>Hint: After working on    this problem for a while, you may find it useful to see the    discussion at    <a href="https://groups.google.com/forum/#!topic/theano-users/NQ9NYLvleGc">this      link</a>.</em></p><p><li> Earlier in the chapter I described a technique for expanding the  training data by applying (small) rotations, skewing, and  translation.  Modify <tt>network3.py</tt> to incorporate all these  techniques.  <em>Note: Unless you have a tremendous amount of    memory, it is not practical to explicitly generate the entire    expanded data set.  So you should consider alternate approaches.</em></p><p><li> Add the ability to load and save networks to <tt>network3.py</tt>.</p><p><li> A shortcoming of the current code is that it provides few  diagnostic tools.  Can you think of any diagnostics to add that  would make it easier to understand to what extent a network is  overfitting?  Add them.</p><p><li> We've used the same initialization procedure for rectified  linear units as for sigmoid (and tanh) neurons.  Our  <a href="chap3.html#weight_initialization">argument for that    initialization</a> was specific to the sigmoid function.  Consider a  network made entirely of rectified linear units (including outputs).  Show that rescaling all the weights in the network by a constant  factor $c > 0$ simply rescales the outputs by a factor $c^{L-1}$,  where $L$ is the number of layers.  How does this change if the  final layer is a softmax?  What do you think of using the sigmoid  initialization procedure for the rectified linear units?  Can you  think of a better initialization procedure?  <em>Note: This is a    very open-ended problem, not something with a simple    self-contained answer.  Still, considering the problem will help    you better understand networks containing rectified linear units.</em></p><p><li> Our  <a href="chap5.html#what's_causing_the_vanishing_gradient_problem_unstable_gradients_in_deep_neural_nets">analysis</a>  of the unstable gradient problem was for sigmoid neurons. How does  the analysis change for networks made up of rectified linear units?  Can you think of a good way of modifying such a network so it  doesn't suffer from the unstable gradient problem?  <em>Note: The    word good in the second part of this makes the problem a research    problem.  It's actually easy to think of ways of making such    modifications.  But I haven't investigated in enough depth to know    of a really good technique.</em></ul></p><p><h3><a name="recent_progress_in_image_recognition"></a><a href="chap6.html#recent_progress_in_image_recognition">Recent progress in image recognition</a></h3></p><p>In 1998, the year MNIST was introduced, it took weeks to train astate-of-the-art workstation to achieve accuracies substantially worsethan those we can achieve using a GPU and less than an hour oftraining. Thus, MNIST is no longer a problem that pushes the limits ofavailable technique; rather, the speed of training means that it is aproblem good for teaching and learning purposes.  Meanwhile, the focusof research has moved on, and modern work involves much morechallenging image recognition problems.  In this section, I brieflydescribe some recent work on image recognition using neural networks.</p><p>The section is different to most of the book.  Through the book I'vefocused on ideas likely to be of lasting interest - ideas such asbackpropagation, regularization, and convolutional networks.  I'vetried to avoid results which are fashionable as I write, but whoselong-term value is unknown. In science, such results are more oftenthan not ephemera which fade and have little lasting impact.  Giventhis, a skeptic might say: "well, surely the recent progress in imagerecognition is an example of such ephemera?  In another two or threeyears, things will have moved on.  So surely these results are only ofinterest to a few specialists who want to compete at the absolutefrontier?  Why bother discussing it?"</p><p>Such a skeptic is right that some of the finer details of recentpapers will gradually diminish in perceived importance.  With thatsaid, the past few years have seen extraordinary improvements usingdeep nets to attack extremely difficult image recognition tasks.Imagine a historian of science writing about computer vision in theyear 2100.  They will identify the years 2011 to 2015 (and probably afew years beyond) as a time of huge breakthroughs, driven by deepconvolutional nets.  That doesn't mean deep convolutional nets willstill be used in 2100, much less detailed ideas such as dropout,rectified linear units, and so on.  But it does mean that an importanttransition is taking place, right now, in the history of ideas.  It'sa bit like watching the discovery of the atom, or the invention ofantibiotics: invention and discovery on a historic scale.  And sowhile we won't dig down deep into details, it's worth getting someidea of the exciting discoveries currently being made.</p><p><strong>The 2012 LRMD paper:</strong> Let me start with a 2012paper*<span class="marginnote">*<a href="http://research.google.com/pubs/pub38115.html">Building    high-level features using large scale unsupervised learning</a>, by  Quoc Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai  Chen, Greg Corrado, Jeff Dean, and Andrew Ng (2012).  Note that the  detailed architecture of the network used in the paper differed in  many details from the deep convolutional networks we've been  studying.  Broadly speaking, however, LRMD is based on many similar  ideas.</span>  from a group of researchers from Stanford and Google.  I'llrefer to this paper as LRMD, after the last names of the first fourauthors. LRMD used a neural network to classify images from<a href="http://www.image-net.org">ImageNet</a>, a very challenging imagerecognition problem.  The 2011 ImageNet data that they used included16 million full color images, in 20 thousand categories.  The imageswere crawled from the open net, and classified by workers fromAmazon's Mechanical Turk service.  Here's a few ImageNetimages*<span class="marginnote">*These are from the 2014 dataset, which is somewhat  changed from 2011.  Qualitatively, however, the dataset is extremely  similar.  Details about ImageNet are available in the original  ImageNet paper,  <a href="http://www.image-net.org/papers/imagenet_cvpr09.pdf">ImageNet:    a large-scale hierarchical image database</a>, by Jia Deng, Wei Dong,  Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei (2009).</span>:</p><p><img src="images/imagenet1.jpg" height="120px"><img src="images/imagenet2.jpg" height="120px"><img src="images/imagenet3.jpg" height="120px"><img src="images/imagenet4.jpg" height="120px"></p><p>These are, respectively, in the categories for beading plane, brownroot rot fungus, scalded milk, and the common roundworm.  If you'relooking for a challenge, I encourage you to visit ImageNet's list of<a href="http://www.image-net.org/synset?wnid=n03489162">hand tools</a>,which distinguishes between beading planes, block planes, chamferplanes, and about a dozen other types of plane, amongst othercategories.  I don't know about you, but I cannot confidentlydistinguish between all these tool types.  This is obviously a muchmore challenging image recognition task than MNIST!  LRMD's networkobtained a respectable $15.8$ percent accuracy for correctlyclassifying ImageNet images.  That may not sound impressive, but itwas a huge improvement over the previous best result of $9.3$ percentaccuracy.  That jump suggested that neural networks might offer apowerful approach to very challenging image recognition tasks, such asImageNet.</p><p><strong>The 2012 KSH paper:</strong> The work of LRMD was followed by a 2012paper of Krizhevsky, Sutskever and Hinton(KSH)*<span class="marginnote">*<a href="http://www.cs.toronto.edu/&#126;fritz/absps/imagenet.pdf">ImageNet    classification with deep convolutional neural networks</a>, by Alex  Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton (2012).</span>. KSHtrained and tested a deep convolutional neural network using arestricted subset of the ImageNet data. The subset they used came froma popular machine learning competition - the ImageNet Large-ScaleVisual Recognition Challenge (ILSVRC).  Using a competition datasetgave them a good way of comparing their approach to other leadingtechniques.  The ILSVRC-2012 training set contained about 1.2 millionImageNet images, drawn from 1,000 categories.  The validation and testsets contained 50,000 and 150,000 images, respectively, drawn from thesame 1,000 categories.</p><p>One difficulty in running the ILSVRC competition is that many ImageNetimages contain multiple objects.  Suppose an image shows a labradorretriever chasing a soccer ball.  The so-called "correct" ImageNetclassification of the image might be as a labrador retriever.  Shouldan algorithm be penalized if it labels the image as a soccer ball?Because of this ambiguity, an algorithm was considered correct if theactual ImageNet classification was among the $5$ classifications thealgorithm considered most likely.  By this top-$5$ criterion, KSH'sdeep convolutional network achieved an accuracy of $84.7$ percent,vastly better than the next-best contest entry, which achieved anaccuracy of $73.8$ percent.  Using the more restrictive metric ofgetting the label exactly right, KSH's network achieved an accuracy of$63.3$ percent.</p><p>It's worth briefly describing KSH's network, since it has inspiredmuch subsequent work.  It's also, as we shall see, closely related tothe networks we trained earlier in this chapter, albeit moreelaborate.  KSH used a deep convolutional neural network, trained ontwo GPUs.  They used two GPUs because the particular type of GPU theywere using (an NVIDIA GeForce GTX 580) didn't have enough on-chipmemory to store their entire network.  So they split the network intotwo parts, partitioned across the two GPUs.</p><p>The KSH network has $7$ layers of hidden neurons.  The first $5$hidden layers are convolutional layers (some with max-pooling), whilethe next $2$ layers are fully-connected layers.  The output layer is a$1,000$-unit softmax layer, corresponding to the $1,000$ imageclasses. Here's a sketch of the network, taken from the KSHpaper*<span class="marginnote">*Thanks to Ilya Sutskever.</span>.  The details are explainedbelow.  Note that many layers are split into $2$ parts, correspondingto the $2$ GPUs.</p><p><img src="images/KSH.jpg" width="600px"></p><p>The input layer contains $3 \times 224 \times 224$ neurons,representing the RGB values for a $224 \times 224$ image.  Recallthat, as mentioned earlier, ImageNet contains images of varyingresolution.  This poses a problem, since a neural network's inputlayer is usually of a fixed size.  KSH dealt with this by rescalingeach image so the shorter side had length $256$. They then cropped outa $256 \times 256$ area in the center of the rescaled image.  Finally,KSH extracted random $224 \times 224$ subimages (and horizontalreflections) from the $256 \times 256$ images.  They did this randomcropping as a way of expanding the training data, and thus reducingoverfitting.  This is particularly helpful in a large network such asKSH's.  It was these $224 \times 224$ images which were used as inputsto the network.  In most cases the cropped image still contains themain object from the uncropped image.</p><p>Moving on to the hidden layers in KSH's network, the first hiddenlayer is a convolutional layer, with a max-pooling step.  It useslocal receptive fields of size $11 \times 11$, and a stride length of$4$ pixels.  There are a total of $96$ feature maps.  The feature mapsare split into two groups of $48$ each, with the first $48$ featuremaps residing on one GPU, and the second $48$ feature maps residing onthe other GPU.  The max-pooling in this and later layers is done in $3\times 3$ regions, but the pooling regions are allowed to overlap, andare just $2$ pixels apart.</p><p>The second hidden layer is also a convolutional layer, with amax-pooling step.  It uses $5 \times 5$ local receptive fields, andthere's a total of $256$ feature maps, split into $128$ on each GPU.Note that the feature maps only use $48$ input channels, not the full$96$ output from the previous layer (as would usually be the case).This is because any single feature map only uses inputs from the sameGPU.  In this sense the network departs from the convolutionalarchitecture we described earlier in the chapter, though obviously thebasic idea is still the same.</p><p>The third, fourth and fifth hidden layers are convolutional layers,but unlike the previous layers, they do not involve max-pooling.Their respectives parameters are: (3) $384$ feature maps, with $3\times 3$ local receptive fields, and $256$ input channels; (4) $384$feature maps, with $3 \times 3$ local receptive fields, and $192$input channels; and (5) $256$ feature maps, with $3 \times 3$ localreceptive fields, and $192$ input channels.  Note that the third layerinvolves some inter-GPU communication (as depicted in the figure) inorder that the feature maps use all $256$ input channels.</p><p>The sixth and seventh hidden layers are fully-connected layers, with$4,096$ neurons in each layer.</p><p>The output layer is a $1,000$-unit softmax layer.</p><p>The KSH network takes advantage of many techniques.  Instead of usingthe sigmoid or tanh activation functions, KSH use rectified linearunits, which sped up training significantly.  KSH's network hadroughly 60 million learned parameters, and was thus, even with thelarge training set, susceptible to overfitting.  To overcome this,they expanded the training set using the random cropping strategy wediscussed above.  They also further addressed overfitting by using avariant of <a href="chap3.html#regularization">l2 regularization</a>, and<a href="chap3.html#other_techniques_for_regularization">dropout</a>.The network itself was trained using<a href="chap3.html#variations_on_stochastic_gradient_descent">momentum-based</a>mini-batch stochastic gradient descent.</p><p>That's an overview of many of the core ideas in the KSH paper.  I'veomitted some details, for which you should look at the paper.  You canalso look at Alex Krizhevsky's<a href="https://code.google.com/p/cuda-convnet/">cuda-convnet</a> (andsuccessors), which contains code implementing many of the ideas.  ATheano-based implementation has also beendeveloped*<span class="marginnote">*<a href="http://arxiv.org/abs/1412.2302">Theano-based    large-scale visual recognition with multiple GPUs</a>, by Weiguang  Ding, Ruoyan Wang, Fei Mao, and Graham Taylor (2014).</span>, with thecode available<a href="https://github.com/uoguelph-mlrg/theano_alexnet">here</a>.  Thecode is recognizably along similar lines to that developed in thischapter, although the use of multiple GPUs complicates thingssomewhat.  The Caffe neural nets framework also includes a version ofthe KSH network, see their<a href="http://caffe.berkeleyvision.org/model_zoo.html">Model Zoo</a> fordetails.</p><p><strong>The 2014 ILSVRC competition:</strong> Since 2012, rapid progresscontinues to be made.  Consider the 2014 ILSVRC competition.  As in2012, it involved a training set of $1.2$ million images, in $1,000$categories, and the figure of merit was whether the top $5$predictions included the correct category.  The winning team, basedprimarily atGoogle*<span class="marginnote">*<a href="http://arxiv.org/abs/1409.4842">Going deeper    with convolutions</a>, by Christian Szegedy, Wei Liu, Yangqing Jia,  Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan,  Vincent Vanhoucke, and Andrew Rabinovich (2014).</span>, used a deepconvolutional network with $22$ layers of neurons.  They called theirnetwork GoogLeNet, as a homage to LeNet-5.  GoogLeNet achieved a top-5accuracy of $93.33$ percent, a giant improvement over the 2013 winner(<a href="http://www.clarifai.com">Clarifai</a>, with $88.3$ percent), andthe 2012 winner (KSH, with $84.7$ percent).</p><p>Just how good is GoogLeNet's $93.33$ percent accuracy?  In 2014 a teamof researchers wrote a survey paper about the ILSVRCcompetition*<span class="marginnote">*<a href="http://arxiv.org/abs/1409.0575">ImageNet    large scale visual recognition challenge</a>, by Olga Russakovsky,  Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,  Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein,  Alexander C. Berg, and Li Fei-Fei (2014).</span>.  One of the questionsthey address is how well humans perform on ILSVRC.  To do this, theybuilt a system which lets humans classify ILSVRC images.  As one ofthe authors, Andrej Karpathy, explains in an informative<a href="http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/">blog  post</a>, it was a lot of trouble to get the humans up to GoogLeNet'sperformance:</p><p><blockquote> ...the task of labeling images with 5 out of 1000  categories quickly turned out to be extremely challenging, even for  some friends in the lab who have been working on ILSVRC and its  classes for a while. First we thought we would put it up on [Amazon  Mechanical Turk]. Then we thought we could recruit paid  undergrads. Then I organized a labeling party of intense labeling  effort only among the (expert labelers) in our lab. Then I developed  a modified interface that used GoogLeNet predictions to prune the  number of categories from 1000 to only about 100. It was still too  hard - people kept missing categories and getting up to ranges of  13-15&#37; error rates. In the end I realized that to get anywhere  competitively close to GoogLeNet, it was most efficient if I sat  down and went through the painfully long training process and the  subsequent careful annotation process myself... The labeling  happened at a rate of about 1 per minute, but this decreased over  time... Some images are easily recognized, while some images (such  as those of fine-grained breeds of dogs, birds, or monkeys) can  require multiple minutes of concentrated effort. I became very good  at identifying breeds of dogs... Based on the sample of images I  worked on, the GoogLeNet classification error turned out to be  6.8&#37;... My own error in the end turned out to be 5.1&#37;,  approximately 1.7&#37; better.  </blockquote></p><p>In other words, an expert human, working painstakingly, was with greateffort able to narrowly beat the deep neural network.  In fact,Karpathy reports that a second human expert, trained on a smallersample of images, was only able to attain a $12.0$ percent top-5 errorrate, significantly below GoogLeNet's performance.  About half theerrors were due to the expert "failing to spot and consider theground truth label as an option".</p><p>These are astonishing results.  Indeed, since this work, several teamshave reported systems whose top-5 error rate is actually <em>better</em>than 5.1&#37;.  This has sometimes been reported in the media as thesystems having better-than-human vision.  While the results aregenuinely exciting, there are many caveats that make it misleading tothink of the systems as having better-than-human vision.  The ILSVRCchallenge is in many ways a rather limited problem - a crawl of theopen web is not necessarily representative of images found inapplications!  And, of course, the top-$5$ criterion is quiteartificial.  We are still a long way from solving the problem of imagerecognition or, more broadly, computer vision.  Still, it's extremelyencouraging to see so much progress made on such a challengingproblem, over just a few years.</p><p><strong>Other activity:</strong> I've focused on ImageNet, but there's aconsiderable amount of other activity using neural nets to do imagerecognition.  Let me briefly describe a few interesting recentresults, just to give the flavour of some current work.</p><p>One encouraging practical set of results comes from a team at Google,who applied deep convolutional networks to the problem of recognizingstreet numbers in Google's Street Viewimagery*<span class="marginnote">*<a href="http://arxiv.org/abs/1312.6082">Multi-digit    Number Recognition from Street View Imagery using Deep    Convolutional Neural Networks</a>, by Ian J. Goodfellow, Yaroslav  Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet (2013).</span>.  Intheir paper, they report detecting and automatically transcribingnearly 100 million street numbers at an accuracy similar to that of ahuman operator.  The system is fast: their system transcribed all ofStreet View's images of street numbers in France in less than an hour!They say: "Having this new dataset significantly increased thegeocoding quality of Google Maps in several countries especially theones that did not already have other sources of good geocoding."  Andthey go on to make the broader claim: "We believe with this model wehave solved [optical character recognition] for short sequences [ofcharacters] for many applications."</p><p>I've perhaps given the impression that it's all a parade ofencouraging results.  Of course, some of the most interesting workreports on fundamental things we don't yet understand.  For instance,a 2013 paper*<span class="marginnote">*<a href="http://arxiv.org/abs/1312.6199">Intriguing    properties of neural networks</a>, by Christian Szegedy, Wojciech  Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow,  and Rob Fergus (2013)</span> showed that deep networks may suffer fromwhat are effectively blind spots.  Consider the lines of images below.On the left is an ImageNet image classified correctly by theirnetwork.  On the right is a slightly perturbed image (the perturbationis in the middle) which is classified <em>incorrectly</em> by thenetwork.  The authors found that there are such "adversarial" imagesfor every sample image, not just a few special ones.</p><p><img src="images/adversarial.jpg"></p><p>This is a disturbing result.  The paper used a network based on thesame code as KSH's network - that is, just the type of network thatis being increasingly widely used.  While such neural networks computefunctions which are, in principle, continuous, results like thissuggest that in practice they're likely to compute functions which arevery nearly discontinuous.  Worse, they'll be discontinuous in waysthat violate our intuition about what is reasonable behavior.  That'sconcerning.  Furthermore, it's not yet well understood what's causingthe discontinuity: is it something about the loss function?  Theactivation functions used?  The architecture of the network?Something else?  We don't yet know.</p><p>Now, these results are not quite as bad as they sound.  Although suchadversarial images are common, they're also unlikely in practice.  Asthe paper notes:</p><p><blockquote>  The existence of the adversarial negatives appears to be in  contradiction with the network’s ability to achieve high  generalization performance. Indeed, if the network can generalize  well, how can it be confused by these adversarial negatives, which  are indistinguishable from the regular examples? The explanation is  that the set of adversarial negatives is of extremely low  probability, and thus is never (or rarely) observed in the test set,  yet it is dense (much like the rational numbers), and so it is found  near virtually every test case.</blockquote></p><p>Nonetheless, it is distressing that we understand neural nets sopoorly that this kind of result should be a recent discovery.  Ofcourse, a major benefit of the results is that they have stimulatedmuch followup work.  For example, one recentpaper*<span class="marginnote">*<a href="http://arxiv.org/abs/1412.1897">Deep Neural    Networks are Easily Fooled: High Confidence Predictions for    Unrecognizable Images</a>, by Anh Nguyen, Jason Yosinski, and Jeff  Clune (2014).</span> shows that given a trained network it's possible togenerate images which look to a human like white noise, but which thenetwork classifies as being in a known category with a very highdegree of confidence.  This is another demonstration that we have along way to go in understanding neural networks and their use in imagerecognition.</p><p>Despite results like this, the overall picture is encouraging.  We'reseeing rapid progress on extremely difficult benchmarks, likeImageNet.  We're also seeing rapid progress in the solution ofreal-world problems, like recognizing street numbers in StreetView.But while this is encouraging it's not enough just to see improvementson benchmarks, or even real-world applications.  There are fundamentalphenomena which we still understand poorly, such as the existence ofadversarial images.  When such fundamental problems are still beingdiscovered (never mind solved), it is premature to say that we're nearsolving the problem of image recognition.  At the same time suchproblems are an exciting stimulus to further work.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p><h3><a name="other_approaches_to_deep_neural_nets"></a><a href="chap6.html#other_approaches_to_deep_neural_nets">Other approaches to deep neural nets</a></h3></p><p>Through this book, we've concentrated on a single problem: classifyingthe MNIST digits.  It's a juicy problem which forced us to understandmany powerful ideas: stochastic gradient descent, backpropagation,convolutional nets, regularization, and more.  But it's also a narrowproblem.  If you read the neural networks literature, you'll run intomany ideas we haven't discussed: recurrent neural networks, Boltzmannmachines, generative models, transfer learning, reinforcementlearning, and so on, on and on $\ldots$ and on!  Neural networks is avast field.  However, many important ideas are variations on ideaswe've already discussed, and can be understood with a little effort.In this section I provide a glimpse of these as yet unseen vistas.The discussion isn't detailed, nor comprehensive - that wouldgreatly expand the book.  Rather, it's impressionistic, an attempt toevoke the conceptual richness of the field, and to relate some ofthose riches to what we've already seen.  Through the section, I'llprovide a few links to other sources, as entrees to learn more.  Ofcourse, many of these links will soon be superseded, and you may wishto search out more recent literature.  That point notwithstanding, Iexpect many of the underlying ideas to be of lasting interest.</p><p><strong>Recurrent neural networks (RNNs):</strong> In the feedforward netswe've been using there is a single input which completely determinesthe activations of all the neurons through the remaining layers.  It'sa very static picture: everything in the network is fixed, with afrozen, crystalline quality to it.  But suppose we allow the elementsin the network to keep changing in a dynamic way.  For instance, thebehaviour of hidden neurons might not just be determined by theactivations in previous hidden layers, but also by the activations atearlier times.  Indeed, a neuron's activation might be determined inpart by its own activation at an earlier time.  That's certainly notwhat happens in a feedforward network.  Or perhaps the activations ofhidden and output neurons won't be determined just by the currentinput to the network, but also by earlier inputs.</p><p>Neural networks with this kind of time-varying behaviour are known as<em>recurrent neural networks</em> or <em>RNNs</em>.  There are manydifferent ways of mathematically formalizing the informal descriptionof recurrent nets given in the last paragraph.  You can get theflavour of some of these mathematical models by glancing at<a href="http://en.wikipedia.org/wiki/Recurrent_neural_network">the  Wikipedia article on RNNs</a>.  As I write, that page lists no fewerthan 13 different models.  But mathematical details aside, the broadidea is that RNNs are neural networks in which there is some notion ofdynamic change over time.  And, not surprisingly, they're particularlyuseful in analysing data or processes that change over time.  Suchdata and processes arise naturally in problems such as speech ornatural language, for example.</p><p>One way RNNs are currently being used is to connect neural networksmore closely to traditional ways of thinking about algorithms, ways ofthinking based on concepts such as Turing machines and (conventional)programming languages.  <a href="http://arxiv.org/abs/1410.4615">A 2014  paper</a> developed an RNN which could take as input acharacter-by-character description of a (very, very simple!) Pythonprogram, and use that description to predict the output.  Informally,the network is learning to "understand" certain Python programs.<a href="http://arxiv.org/abs/1410.5401">A second paper, also from 2014</a>,used RNNs as a starting point to develop what they called a neuralTuring machine (NTM).  This is a universal computer whose entirestructure can be trained using gradient descent.  They trained theirNTM to infer algorithms for several simple problems, such as sortingand copying.</p><p>As it stands, these are extremely simple toy models.  Learning toexecute the Python program <tt>print(398345+42598)</tt> doesn't make anetwork into a full-fledged Python interpreter!  It's not clear howmuch further it will be possible to push the ideas.  Still, theresults are intriguing.  Historically, neural networks have done wellat pattern recognition problems where conventional algorithmicapproaches have trouble.  Vice versa, conventional algorithmicapproaches are good at solving problems that neural nets aren't sogood at.  No-one today implements a web server or a database programusing a neural network!  It'd be great to develop unified models thatintegrate the strengths of both neural networks and more traditionalapproaches to algorithms.  RNNs and ideas inspired by RNNs may help usdo that.</p><p>RNNs have also been used in recent years to attack many otherproblems.  They've been particularly useful in speech recognition.Approaches based on RNNs have, for example,<a href="http://arxiv.org/abs/1303.5778">set records for the accuracy of  phoneme recognition</a>.  They've also been used to develop<a href="http://www.fit.vutbr.cz/&#126;imikolov/rnnlm/thesis.pdf">improved  models of the language people use while speaking</a>.  Better languagemodels help disambiguate utterances that otherwise sound alike.  Agood language model will, for example, tell us that "to infinity andbeyond" is much more likely than "two infinity and beyond", despitethe fact that the phrases sound identical.  RNNs have been used to setnew records for certain language benchmarks.</p><p>This work is, incidentally, part of a broader use of deep neural netsof all types, not just RNNs, in speech recognition.  For example, anapproach based on deep nets has achieved<a href="http://arxiv.org/abs/1309.1501">outstanding results on large  vocabulary continuous speech recognition</a>.  And another system basedon deep nets has been deployed in<a href="http://www.wired.com/2013/02/android-neural-network/">Google's  Android operating system</a> (for related technical work, see<a href="http://research.google.com/pubs/VincentVanhoucke.html">Vincent  Vanhoucke's 2012-2015 papers</a>).</p><p>I've said a little about what RNNs can do, but not so much about howthey work.  It perhaps won't surprise you to learn that many of theideas used in feedforward networks can also be used in RNNs.  Inparticular, we can train RNNs using straightforward modifications togradient descent and backpropagation.  Many other ideas used infeedforward nets, ranging from regularization techniques toconvolutions to the activation and cost functions used, are alsouseful in recurrent nets.  And so many of the techniques we'vedeveloped in the book can be adapted for use with RNNs.</p><p></p><p></p><p></p><p><strong>Long short-term memory units (LSTMs):</strong> One challenge affectingRNNs is that early models turned out to be very difficult to train,harder even than deep feedforward networks.  The reason is theunstable gradient problem discussed in <a href="chap5.html">Chapter 5</a>.Recall that the usual manifestation of this problem is that thegradient gets smaller and smaller as it is propagated back throughlayers.  This makes learning in early layers extremely slow.  Theproblem actually gets worse in RNNs, since gradients aren't justpropagated backward through layers, they're propagated backwardthrough time.  If the network runs for a long time that can make thegradient extremely unstable and hard to learn from.  Fortunately, it'spossible to incorporate an idea known as long short-term memory units(LSTMs) into RNNs.  The units were introduced by<a href="http://dx.doi.org/10.1162/neco.1997.9.8.1735">Hochreiter and  Schmidhuber in 1997</a> with the explicit purpose of helping addressthe unstable gradient problem.  LSTMs make it much easier to get goodresults when training RNNs, and many recent papers (including manythat I linked above) make use of LSTMs or related ideas.</p><p><strong>Deep belief nets, generative models, and Boltzmann machines:</strong>Modern interest in deep learning began in 2006, with papers explaininghow to train a type of neural network known as a <em>deep belief  network</em> (DBN)*<span class="marginnote">*See  <a href="http://www.cs.toronto.edu/&#126;hinton/absps/fastnc.pdf">A fast    learning algorithm for deep belief nets</a>, by Geoffrey Hinton,  Simon Osindero, and Yee-Whye Teh (2006), as well as the related work  in  <a href="http://www.sciencemag.org/content/313/5786/504.short">Reducing    the dimensionality of data with neural networks</a>, by Geoffrey  Hinton and Ruslan Salakhutdinov (2006).</span>.  DBNs were influential forseveral years, but have since lessened in popularity, while modelssuch as feedforward networks and recurrent neural nets have becomefashionable.  Despite this, DBNs have several properties that makethem interesting.</p><p>One reason DBNs are interesting is that they're an example of what'scalled a <em>generative model</em>.  In a feedforward network, wespecify the input activations, and they determine the activations ofthe feature neurons later in the network.  A generative model like aDBN can be used in a similar way, but it's also possible to specifythe values of some of the feature neurons and then "run the networkbackward", generating values for the input activations.  Moreconcretely, a DBN trained on images of handwritten digits can(potentially, and with some care) also be used to generate images thatlook like handwritten digits.  In other words, the DBN would in somesense be learning to write.  In this, a generative model is much likethe human brain: not only can it read digits, it can also write them.In Geoffrey Hinton's memorable phrase,<a href="http://www.sciencedirect.com/science/article/pii/S0079612306650346">to  recognize shapes, first learn to generate images</a>.</p><p>A second reason DBNs are interesting is that they can do unsupervisedand semi-supervised learning.  For instance, when trained with imagedata, DBNs can learn useful features for understanding other images,even if the training images are unlabelled.  And the ability to dounsupervised learning is extremely interesting both for fundamentalscientific reasons, and - if it can be made to work well enough -for practical applications.</p><p>Given these attractive features, why have DBNs lessened in popularityas models for deep learning?  Part of the reason is that models suchas feedforward and recurrent nets have achieved many spectacularresults, such as their breakthroughs on image and speech recognitionbenchmarks.  It's not surprising and quite right that there's now lotsof attention being paid to these models.  There's an unfortunatecorollary, however.  The marketplace of ideas often functions in awinner-take-all fashion, with nearly all attention going to thecurrent fashion-of-the-moment in any given area.  It can becomeextremely difficult for people to work on momentarily unfashionableideas, even when those ideas are obviously of real long-term interest.My personal opinion is that DBNs and other generative models likelydeserve more attention than they are currently receiving.  And I won'tbe surprised if DBNs or a related model one day surpass the currentlyfashionable models.  For an introduction to DBNs, see<a href="http://www.scholarpedia.org/article/Deep_belief_networks">this  overview</a>.  I've also found<a href="http://www.cs.toronto.edu/&#126;hinton/absps/guideTR.pdf">this  article</a> helpful.  It isn't primarily about deep belief nets,<em>per se</em>, but does contain much useful information aboutrestricted Boltzmann machines, which are a key component of DBNs.</p><p><strong>Other ideas:</strong> What else is going on in neural networks anddeep learning?  Well, there's a huge amount of other fascinating work.Active areas of research include using neural networks to do<a href="http://machinelearning.org/archive/icml2008/papers/391.pdf">natural  language processing</a> (see <a href="http://arxiv.org/abs/1103.0398">also  this informative review paper</a>),<a href="assets/MachineTranslation.pdf">machine translation</a>, as well asperhaps more surprising applications such as<a href="http://yann.lecun.com/exdb/publis/pdf/humphrey-jiis-13.pdf">music  informatics</a>.  There are, of course, many other areas too.  In manycases, having read this book you should be able to begin followingrecent work, although (of course) you'll need to fill in gaps inpresumed background knowledge.</p><p>Let me finish this section by mentioning a particularly fun paper.  Itcombines deep convolutional networks with a technique known asreinforcement learning in order to learn to<a href="http://www.cs.toronto.edu/&#126;vmnih/docs/dqn.pdf">play video games  well</a> (see also<a href="http://www.nature.com/nature/journal/v518/n7540/abs/nature14236.html">this  followup</a>).  The idea is to use the convolutional network tosimplify the pixel data from the game screen, turning it into asimpler set of features, which can be used to decide which action totake: "go left", "go down", "fire", and so on.  What isparticularly interesting is that a single network learned to playseven different classic video games pretty well, outperforming humanexperts on three of the games.  Now, this all sounds like a stunt, andthere's no doubt the paper was well marketed, with the title "PlayingAtari with reinforcement learning".  But looking past the surfacegloss, consider that this system is taking raw pixel data - itdoesn't even know the game rules! - and from that data learning todo high-quality decision-making in several very different and veryadversarial environments, each with its own complex set of rules.That's pretty neat.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p> </p><p></p><p><h3><a name="on_the_future_of_neural_networks"></a><a href="chap6.html#on_the_future_of_neural_networks">On the future of neural networks</a></h3></p><p><strong>Intention-driven user interfaces:</strong> There's an old joke inwhich an impatient professor tells a confused student: "don't listento what I say; listen to what I <em>mean</em>".  Historically,computers have often been, like the confused student, in the darkabout what their users mean.  But this is changing.  I still remembermy surprise the first time I misspelled a Google search query, only tohave Google say "Did you mean [corrected query]?" and to offer thecorresponding search results.  Google CEO Larry Page<a href="http://googleblog.blogspot.ca/2012/08/building-search-engine-of-future-one.html">once  described the perfect search engine as understanding exactly what  [your queries] mean and giving you back exactly what you want</a>.</p><p>This is a vision of an <em>intention-driven user interface</em>.  Inthis vision, instead of responding to users' literal queries, searchwill use machine learning to take vague user input, discern preciselywhat was meant, and take action on the basis of those insights.</p><p>The idea of intention-driven interfaces can be applied far morebroadly than search.  Over the next few decades, thousands ofcompanies will build products which use machine learning to make userinterfaces that can tolerate imprecision, while discerning and actingon the user's true intent.  We're already seeing early examples ofsuch intention-driven interfaces: Apple's Siri; Wolfram Alpha; IBM'sWatson; systems which can<a href="http://arxiv.org/abs/1411.4555">annotate photos and videos</a>; andmuch more.</p><p>Most of these products will fail. Inspired user interface design ishard, and I expect many companies will take powerful machine learningtechnology and use it to build insipid user interfaces.  The bestmachine learning in the world won't help if your user interfaceconcept stinks.  But there will be a residue of products whichsucceed.  Over time that will cause a profound change in how we relateto computers.  Not so long ago - let's say, 2005 - users took itfor granted that they needed precision in most interactions withcomputers.  Indeed, computer literacy to a great extent meantinternalizing the idea that computers are extremely literal; a singlemisplaced semi-colon may completely change the nature of aninteraction with a computer.  But over the next few decades I expectwe'll develop many successful intention-driven user interfaces, andthat will dramatically change what we expect when interacting withcomputers.</p><p><strong>Machine learning, data science, and the virtuous circle of  innovation:</strong> Of course, machine learning isn't just being used tobuild intention-driven interfaces.  Another notable application is indata science, where machine learning is used to find the "knownunknowns" hidden in data.  This is already a fashionable area, andmuch has been written about it, so I won't say much.  But I do want tomention one consequence of this fashion that is not so often remarked:over the long run it's possible the biggest breakthrough in machinelearning won't be any single conceptual breakthrough. Rather, thebiggest breakthrough will be that machine learning research becomesprofitable, through applications to data science and other areas.  Ifa company can invest 1 dollar in machine learning research and get 1dollar and 10 cents back reasonably rapidly, then a lot of money willend up in machine learning research.  Put another way, machinelearning is an engine driving the creation of several major newmarkets and areas of growth in technology. The result will be largeteams of people with deep subject expertise, and with access toextraordinary resources.  That will propel machine learning furtherforward, creating more markets and opportunities, a virtuous circle ofinnovation.</p><p><strong>The role of neural networks and deep learning:</strong> I've beentalking broadly about machine learning as a creator of newopportunities for technology.  What will be the specific role ofneural networks and deep learning in all this?</p><p>To answer the question, it helps to look at history.  Back in the1980s there was a great deal of excitement and optimism about neuralnetworks, especially after backpropagation became widely known.  Thatexcitement faded, and in the 1990s the machine learning baton passedto other techniques, such as support vector machines.  Today, neuralnetworks are again riding high, setting all sorts of records,defeating all comers on many problems.  But who is to say thattomorrow some new approach won't be developed that sweeps neuralnetworks away again?  Or perhaps progress with neural networks willstagnate, and nothing will immediately arise to take their place?</p><p>For this reason, it's much easier to think broadly about the future ofmachine learning than about neural networks specifically.  Part of theproblem is that we understand neural networks so poorly.  Why is itthat neural networks can generalize so well?  How is it that theyavoid overfitting as well as they do, given the very large number ofparameters they learn?  Why is it that stochastic gradient descentworks as well as it does?  How well will neural networks perform asdata sets are scaled?  For instance, if ImageNet was expanded by afactor of $10$, would neural networks' performance improve more orless than other machine learning techniques?  These are all simple,fundamental questions.  And, at present, we understand the answers tothese questions very poorly.  While that's the case, it's difficult tosay what role neural networks will play in the future of machinelearning.</p><p>I will make one prediction: I believe deep learning is here to stay.The ability to learn hierarchies of concepts, building up multiplelayers of abstraction, seems to be fundamental to making sense of theworld.  This doesn't mean tomorrow's deep learners won't be radicallydifferent than today's.  We could see major changes in the constituentunits used, in the architectures, or in the learning algorithms.Those changes may be dramatic enough that we no longer think of theresulting systems as neural networks.  But they'd still be doing deeplearning.</p><p><a name="AI"></a></p><p><strong>Will neural networks and deep learning soon lead to artificial  intelligence?</strong>  In this book we've focused on using neural nets todo specific tasks, such as classifying images.  Let's broaden ourambitions, and ask: what about general-purpose thinking computers?Can neural networks and deep learning help us solve the problem of(general) artificial intelligence (AI)?  And, if so, given the rapidrecent progress of deep learning, can we expect general AI any timesoon?</p><p>Addressing these questions comprehensively would take a separate book.Instead, let me offer one observation.  It's based on an idea known as<a href="http://en.wikipedia.org/wiki/Conway%27s_law">Conway's law</a>:<blockquote>  Any organization that designs a system... will inevitably produce a  design whose structure is a copy of the organization's communication  structure.</blockquote> So, for example, Conway's law suggests that the design of a Boeing 747aircraft will mirror the extended organizational structure of Boeingand its contractors at the time the 747 was designed.  Or for asimple, specific example, consider a company building a complexsoftware application.  If the application's dashboard is supposed tobe integrated with some machine learning algorithm, the personbuilding the dashboard better be talking to the company's machinelearning expert.  Conway's law is merely that observation, writ large.</p><p>Upon first hearing Conway's law, many people respond either "Well,isn't that banal and obvious?" or "Isn't that wrong?"  Let me startwith the objection that it's wrong.  As an instance of this objection,consider the question: where does Boeing's accounting department showup in the design of the 747?  What about their janitorial department?Their internal catering?  And the answer is that these parts of theorganization probably don't show up explicitly anywhere in the 747.So we should understand Conway's law as referring only to those partsof an organization concerned explicitly with design and engineering.</p><p>What about the other objection, that Conway's law is banal andobvious?  This may perhaps be true, but I don't think so, fororganizations too often act with disregard for Conway's law.  Teamsbuilding new products are often bloated with legacy hires or,contrariwise, lack a person with some crucial expertise.  Think of allthe products which have useless complicating features.  Or think ofall the products which have obvious major deficiencies - e.g., aterrible user interface.  Problems in both classes are often caused bya mismatch between the team that was needed to produce a good product,and the team that was actually assembled.  Conway's law may beobvious, but that doesn't mean people don't routinely ignore it.</p><p>Conway's law applies to the design and engineering of systems where westart out with a pretty good understanding of the likely constituentparts, and how to build them.  It can't be applied directly to thedevelopment of artificial intelligence, because AI isn't (yet) such aproblem: we don't know what the constituent parts are.  Indeed, we'renot even sure what basic questions to be asking.  In others words, atthis point AI is more a problem of science than of engineering.Imagine beginning the design of the 747 without knowing about jetengines or the principles of aerodynamics.  You wouldn't know whatkinds of experts to hire into your organization.  As Wernher von Braunput it, "basic research is what I'm doing when I don't know what I'mdoing".  Is there a version of Conway's law that applies to problemswhich are more science than engineering?</p><p>To gain insight into this question, consider the history of medicine.In the early days, medicine was the domain of practitioners like Galenand Hippocrates, who studied the entire body.  But as our knowledgegrew, people were forced to specialize.  We discovered many deep newideas*<span class="marginnote">*My apologies for overloading "deep".  I won't define  "deep ideas" precisely, but loosely I mean the kind of idea which  is the basis for a rich field of enquiry.  The backpropagation  algorithm and the germ theory of disease are both good examples.</span>:think of things like the germ theory of disease, for instance, or theunderstanding of how antibodies work, or the understanding that theheart, lungs, veins and arteries form a complete cardiovascularsystem.  Such deep insights formed the basis for subfields such asepidemiology, immunology, and the cluster of inter-linked fieldsaround the cardiovascular system.  And so the structure of ourknowledge has shaped the social structure of medicine.  This isparticularly striking in the case of immunology: realizing the immunesystem exists and is a system worthy of study is an extremelynon-trivial insight.  So we have an entire field of medicine - withspecialists, conferences, even prizes, and so on - organized aroundsomething which is not just invisible, it's arguably not a distinctthing at all.</p><p>This is a common pattern that has been repeated in manywell-established sciences: not just medicine, but physics,mathematics, chemistry, and others.  The fields start out monolithic,with just a few deep ideas.  Early experts can master all those ideas.But as time passes that monolithic character changes.  We discovermany deep new ideas, too many for any one person to really master.  Asa result, the social structure of the field re-organizes and dividesaround those ideas.  Instead of a monolith, we have fields withinfields within fields, a complex, recursive, self-referential socialstructure, whose organization mirrors the connections between ourdeepest insights.  <em>And so the structure of our knowledge shapes  the social organization of science.  But that social shape in turn  constrains and helps determine what we can discover.</em>  This is the  scientific analogue of Conway's law. </p><p></p><p>So what's this got to do with deep learning or AI?</p><p>Well, since the early days of AI there have been arguments about itthat go, on one side, "Hey, it's not going to be so hard, we've got[super-special weapon] on our side", countered by "[super-specialweapon] won't be enough".  Deep learning is the latest super-specialweapon I've heard used in such arguments*<span class="marginnote">*Interestingly, often  not by leading experts in deep learning, who have been quite  restrained.  See, for example, this  <a href="https://www.facebook.com/yann.lecun/posts/10152348155137143">thoughtful    post</a> by Yann LeCun. This is a difference from many earlier  incarnations of the argument.</span>; earlier versions of the argumentused logic, or Prolog, or expert systems, or whatever the mostpowerful technique of the day was.  The problem with such arguments isthat they don't give you any good way of saying just how powerful anygiven candidate super-special weapon is.  Of course, we've just spenta chapter reviewing evidence that deep learning can solve extremelychallenging problems.  It certainly looks very exciting and promising.But that was also true of systems like Prolog or<a href="http://en.wikipedia.org/wiki/Eurisko">Eurisko</a> or expert systemsin their day.  And so the mere fact that a set of ideas looks verypromising doesn't mean much.  How can we tell if deep learning istruly different from these earlier ideas?  Is there some way ofmeasuring how powerful and promising a set of ideas is?  Conway's lawsuggests that as a rough and heuristic proxy metric we can evaluatethe complexity of the social structure associated to those ideas.</p><p>So, there are two questions to ask.  First, how powerful a set ofideas are associated to deep learning, according to this metric ofsocial complexity?  Second, how powerful a theory will we need, inorder to be able to build a general artificial intelligence?</p><p>As to the first question: when we look at deep learning today, it's anexciting and fast-paced but also relatively monolithic field.  Thereare a few deep ideas, and a few main conferences, with substantialoverlap between several of the conferences.  And there is paper afterpaper leveraging the same basic set of ideas: using stochasticgradient descent (or a close variation) to optimize a cost function.It's fantastic those ideas are so successful.  But what we don't yetsee is lots of well-developed subfields, each exploring their own setsof deep ideas, pushing deep learning in many directions.  And so,according to the metric of social complexity, deep learning is, ifyou'll forgive the play on words, still a rather shallow field.  It'sstill possible for one person to master most of the deepest ideas inthe field.</p><p>On the second question: how complex and powerful a set of ideas willbe needed to obtain AI?  Of course, the answer to this question is:no-one knows for sure.  But in the <a href="sai.html">appendix</a> I examinesome of the existing evidence on this question.  I conclude that, evenrather optimistically, it's going to take many, many deep ideas tobuild an AI.  And so Conway's law suggests that to get to such a pointwe will necessarily see the emergence of many interrelatingdisciplines, with a complex and surprising structure mirroring thestructure in our deepest insights.  We don't yet see this rich socialstructure in the use of neural networks and deep learning.  And so, Ibelieve that we are several decades (at least) from using deeplearning to develop general AI.</p><p>I've gone to a lot of trouble to construct an argument which istentative, perhaps seems rather obvious, and which has an indefiniteconclusion.  This will no doubt frustrate people who crave certainty.Reading around online, I see many people who loudly assert verydefinite, very strongly held opinions about AI, often on the basis offlimsy reasoning and non-existent evidence.  My frank opinion is this:it's too early to say.  As the old joke goes, if you ask a scientisthow far away some discovery is and they say "10 years" (or more),what they mean is "I've got no idea".  AI, like controlled fusionand a few other technologies, has been 10 years away for 60 plusyears.  On the flipside, what we definitely do have in deep learningis a powerful technique whose limits have not yet been found, and manywide-open fundamental problems.  That's an exciting creativeopportunity.</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></div><div class="footer"> <span class="left_footer"> In academic work,
please cite this book as: Michael A. Nielsen, "Neural Networks and
Deep Learning", Determination Press, 2015

<br/>
<br/>

This work is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-nc/3.0/deed.en_GB"
style="color: #eee;">Creative Commons Attribution-NonCommercial 3.0
Unported License</a>.  This means you're free to copy, share, and
build on this book, but not to sell it.  If you're interested in
commercial use, please <a
href="mailto:mn@michaelnielsen.org">contact me</a>.
</span>
<span class="right_footer">
Last update: Thu Dec 26 15:26:33 2019
<br/>
<br/>
<br/>
<a rel="license" href="http://creativecommons.org/licenses/by-nc/3.0/deed.en_GB"><img alt="Creative Commons Licence" style="border-width:0" src="http://i.creativecommons.org/l/by-nc/3.0/88x31.png" /></a>
</span>
</div>
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-44208967-1', 'neuralnetworksanddeeplearning.com');
  ga('send', 'pageview');

</script>
</body>
</html>